[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:53 dist-test-slave-zqrn krb5kdc[18154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:53 dist-test-slave-zqrn krb5kdc[18154](info): set up 2 sockets
Sep 27 04:02:53 dist-test-slave-zqrn krb5kdc[18154](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 04:02:54.082664 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:02:54.210606 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:42427
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:42427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:54.292793 18167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:54.293027 18167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:54.293052 18167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:54.294209 18167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:54.294245 18167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:54.294260 18167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:54.294273 18167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:54.294286 18167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:54.295797 18167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:42427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:42427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:54.296010 18167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:54.296187 18167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:54.301239 18175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:54.301304 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:54.301239 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:54.301296 18167 server_base.cc:1034] running on GCE node
I20260927 04:02:54.301911 18167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:54.302225 18167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:54.303397 18167 hybrid_clock.cc:648] HybridClock initialized: now 1790481774303379 us; error 44 us; skew 500 ppm
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:54.306052 18167 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:02:54.306461 18167 webserver.cc:460] Webserver started at http://127.17.184.254:39011/ using document root <none> and password file <none>
I20260927 04:02:54.306659 18167 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:54.306701 18167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:54.306843 18167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:54.308027 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "d5836eddb63e43e5904c8a74a8dd9e60"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "42347465bc8518c1f8e160580b5758cb"
server_key_iv: "3e7e188977923e2716417993dfb38754"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.308410 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "d5836eddb63e43e5904c8a74a8dd9e60"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "42347465bc8518c1f8e160580b5758cb"
server_key_iv: "3e7e188977923e2716417993dfb38754"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.309767 18167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:02:54.310541 18182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.310722 18167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:54.310823 18167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "d5836eddb63e43e5904c8a74a8dd9e60"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "42347465bc8518c1f8e160580b5758cb"
server_key_iv: "3e7e188977923e2716417993dfb38754"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.310923 18167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:54.340335 18167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:54.342489 18167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:54.342675 18167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:54.346648 18167 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:42427
I20260927 04:02:54.346688 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:42427 every 8 connection(s)
I20260927 04:02:54.347083 18167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:54.348033 18235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:54.350291 18235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60: Bootstrap starting.
I20260927 04:02:54.350847 18235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:54.351177 18235 log.cc:824] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:54.351936 18235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60: No bootstrap required, opened a new log
I20260927 04:02:54.353653 18235 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5836eddb63e43e5904c8a74a8dd9e60" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42427 } }
I20260927 04:02:54.353813 18235 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:54.353842 18235 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5836eddb63e43e5904c8a74a8dd9e60, State: Initialized, Role: FOLLOWER
I20260927 04:02:54.354007 18235 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5836eddb63e43e5904c8a74a8dd9e60" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42427 } }
I20260927 04:02:54.354116 18235 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:54.354143 18235 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:54.354225 18235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:54.354796 18235 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5836eddb63e43e5904c8a74a8dd9e60" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42427 } }
I20260927 04:02:54.354938 18235 leader_election.cc:302] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5836eddb63e43e5904c8a74a8dd9e60; no voters:
I20260927 04:02:54.355139 18235 leader_election.cc:288] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:54.355289 18238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:54.355417 18235 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:54.355444 18238 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [term 1 LEADER]: Becoming Leader. State: Replica: d5836eddb63e43e5904c8a74a8dd9e60, State: Running, Role: LEADER
I20260927 04:02:54.355535 18238 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5836eddb63e43e5904c8a74a8dd9e60" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42427 } }
I20260927 04:02:54.356004 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 18167
I20260927 04:02:54.356127 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:02:54.356104 18240 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5836eddb63e43e5904c8a74a8dd9e60. Latest consensus state: current_term: 1 leader_uuid: "d5836eddb63e43e5904c8a74a8dd9e60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5836eddb63e43e5904c8a74a8dd9e60" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42427 } } }
I20260927 04:02:54.356191 18240 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:54.356240 18239 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5836eddb63e43e5904c8a74a8dd9e60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5836eddb63e43e5904c8a74a8dd9e60" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42427 } } }
I20260927 04:02:54.356310 18239 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60 [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:54.356567 18247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:54.356957 18147 external_mini_cluster.cc:1428] Setting key 681e5e4f96af32ebd2cb4a72217d72e1
I20260927 04:02:54.357100 18247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:54.358886 18247 catalog_manager.cc:1347] Generated new cluster ID: a03749a489424282890f5e3d570dfd1e
I20260927 04:02:54.358974 18247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:54.372393 18247 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:54.373178 18247 catalog_manager.cc:1504] Loading token signing keys...
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:54.379755 18247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5836eddb63e43e5904c8a74a8dd9e60: Generated new TSK 0
I20260927 04:02:54.379956 18247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:02:54.384887 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.357870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32998 (local address 127.17.184.254:42427)
0927 04:02:54.358031 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:54.358040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:54.358622 (+ 582us) server_negotiation.cc:407] Connection header received
0927 04:02:54.369117 (+ 10495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:54.369127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:54.369196 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:54.369377 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:54.371412 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.373090 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.374593 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.375351 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.375479 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.380267 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:54.380272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:54.380272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:54.380283 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:54.382546 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.383166 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.383170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.383171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.383285 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.383655 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.383656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.383656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.383908 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:54.384024 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:54.384453 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 04:02:54.384583 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:02:54.414597 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42427
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:54.498126 18261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:54.498329 18261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:54.498353 18261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:54.499644 18261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:54.499688 18261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:54.499742 18261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:02:54.501185 18261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:54.501441 18261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:54.501693 18261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:54.506889 18266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:54.506953 18261 server_base.cc:1034] running on GCE node
W20260927 04:02:54.506827 18267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:54.506914 18269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:54.507215 18261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:54.507426 18261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:54.508597 18261 hybrid_clock.cc:648] HybridClock initialized: now 1790481774508557 us; error 54 us; skew 500 ppm
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:54.511233 18261 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:02:54.511683 18261 webserver.cc:460] Webserver started at http://127.17.184.193:38491/ using document root <none> and password file <none>
I20260927 04:02:54.511919 18261 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:54.511963 18261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:54.512108 18261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:54.513192 18261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "7ef62014ddf04e8c9603e03eb2531528"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "936c12b0173b1d2aeea3f412c224166c"
server_key_iv: "df03fb171b24100bdec5e1a7d82ce898"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.513583 18261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "7ef62014ddf04e8c9603e03eb2531528"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "936c12b0173b1d2aeea3f412c224166c"
server_key_iv: "df03fb171b24100bdec5e1a7d82ce898"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.515050 18261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:02:54.515899 18276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.516111 18261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:54.516219 18261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "7ef62014ddf04e8c9603e03eb2531528"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "936c12b0173b1d2aeea3f412c224166c"
server_key_iv: "df03fb171b24100bdec5e1a7d82ce898"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.516299 18261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:54.533087 18261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:54.535518 18261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:54.535671 18261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:54.535945 18261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:54.536266 18261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:54.536322 18261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.536365 18261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:54.536393 18261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.542541 18261 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:41055
I20260927 04:02:54.542717 18389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:41055 every 8 connection(s)
I20260927 04:02:54.543000 18261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:54.549682 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 18261
I20260927 04:02:54.549762 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:54.549953 18147 external_mini_cluster.cc:1428] Setting key b946389a3d113700c489de38e80e3c46
I20260927 04:02:54.553536 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.543870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:58123 (local address 127.17.184.254:42427)
0927 04:02:54.543918 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:54.543921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:54.544100 (+ 179us) server_negotiation.cc:407] Connection header received
0927 04:02:54.544663 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:54.544664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:54.544678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:54.544725 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:54.545707 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.546230 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.547305 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.547525 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.547612 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.549996 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:54.549997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:54.549998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:54.550011 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:54.551973 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.552488 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.552489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.552490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.552557 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.552837 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.552838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.552839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.552961 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:54.553037 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:54.553345 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 04:02:54.553442 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:02:54.553730 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.543805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42427 (local address 127.17.184.193:58123)
0927 04:02:54.543961 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:54.544012 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:54.544541 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:54.544798 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:54.544802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:54.545400 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:54.545594 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:54.545598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.546324 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.546325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:54.547203 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:54.547205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.547622 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.547623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:54.547744 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.548272 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:54.548280 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:54.549884 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:54.552077 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:54.552083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:54.552085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:54.552402 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:54.552629 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:54.552630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:54.552630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:54.552753 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:54.553028 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:54.553035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:54.553192 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 04:02:54.553309 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 04:02:54.553478 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260927 04:02:54.554446 18390 heartbeater.cc:347] Connected to a master server at 127.17.184.254:42427
I20260927 04:02:54.554554 18390 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:54.554733 18390 heartbeater.cc:511] Master 127.17.184.254:42427 requested a full tablet report, sending...
I20260927 04:02:54.555398 18199 ts_manager.cc:194] Registered new tserver with Master: 7ef62014ddf04e8c9603e03eb2531528 (127.17.184.193:41055)
I20260927 04:02:54.556538 18199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:58123
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:02:54.578425 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42427
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:54.664345 18395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:54.664585 18395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:54.664625 18395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:54.665802 18395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:54.665867 18395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:54.665913 18395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:02:54.667454 18395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:54.667738 18395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:54.667986 18395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:54.673537 18400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:54.673518 18403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:54.673509 18401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:54.673761 18395 server_base.cc:1034] running on GCE node
I20260927 04:02:54.674062 18395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:54.674345 18395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:54.675494 18395 hybrid_clock.cc:648] HybridClock initialized: now 1790481774675479 us; error 28 us; skew 500 ppm
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:54.678175 18395 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:02:54.678619 18395 webserver.cc:460] Webserver started at http://127.17.184.194:40583/ using document root <none> and password file <none>
I20260927 04:02:54.678855 18395 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:54.678947 18395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:54.679096 18395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:54.680166 18395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "ded682afcc224901bbba7694a25969b1"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "e118530e76f215dfc134ab17b61e5fcc"
server_key_iv: "dcb646d67053a9d31f9ace64df023231"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.680544 18395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "ded682afcc224901bbba7694a25969b1"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "e118530e76f215dfc134ab17b61e5fcc"
server_key_iv: "dcb646d67053a9d31f9ace64df023231"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.681952 18395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:54.682725 18410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.682955 18395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:54.683039 18395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "ded682afcc224901bbba7694a25969b1"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "e118530e76f215dfc134ab17b61e5fcc"
server_key_iv: "dcb646d67053a9d31f9ace64df023231"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.683091 18395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:54.697811 18395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:54.700057 18395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:54.700167 18395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:54.700354 18395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:54.700688 18395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:54.700723 18395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.700745 18395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:54.700765 18395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.706741 18395 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:36843
I20260927 04:02:54.706835 18523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:36843 every 8 connection(s)
I20260927 04:02:54.707168 18395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:54.714038 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 18395
I20260927 04:02:54.714149 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:54.714366 18147 external_mini_cluster.cc:1428] Setting key cb3279245cd83ff5eb1e813d9c3475e6
I20260927 04:02:54.717180 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.708005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:41785 (local address 127.17.184.254:42427)
0927 04:02:54.708091 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:54.708093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:54.708185 (+ 92us) server_negotiation.cc:407] Connection header received
0927 04:02:54.708704 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:54.708705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:54.708715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:54.708818 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:54.709874 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.710381 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.711495 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.711692 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.711796 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.713969 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:54.713970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:54.713971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:54.713984 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:54.715626 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.716138 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.716138 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.716139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.716191 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.716472 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.716473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.716473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.716629 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:54.716701 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:54.716971 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 04:02:54.717059 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 04:02:54.717247 18526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.707927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42427 (local address 127.17.184.194:41785)
0927 04:02:54.708083 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:54.708120 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:54.708612 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:54.708883 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:54.708886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:54.709520 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:54.709721 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:54.709728 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.710475 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.710476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:54.711402 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:54.711404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.711785 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.711786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:54.711895 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.712417 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:54.712426 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:54.713854 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:54.715713 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:54.715720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:54.715722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:54.716059 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:54.716256 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:54.716257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:54.716257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:54.716402 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:54.716707 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:54.716712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:54.716842 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 04:02:54.716928 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:02:54.717055 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260927 04:02:54.718024 18524 heartbeater.cc:347] Connected to a master server at 127.17.184.254:42427
I20260927 04:02:54.718132 18524 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:54.718336 18524 heartbeater.cc:511] Master 127.17.184.254:42427 requested a full tablet report, sending...
I20260927 04:02:54.718734 18199 ts_manager.cc:194] Registered new tserver with Master: ded682afcc224901bbba7694a25969b1 (127.17.184.194:36843)
I20260927 04:02:54.719321 18199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:41785
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:02:54.742599 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42427
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:54.825541 18529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:54.825799 18529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:54.825845 18529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:54.827066 18529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:54.827132 18529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:54.827178 18529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:02:54.828652 18529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42427
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:54.828920 18529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:54.829155 18529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:54.835668 18534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:54.835671 18537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:54.835954 18535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:54.836004 18529 server_base.cc:1034] running on GCE node
I20260927 04:02:54.836272 18529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:54.836580 18529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:54.837944 18529 hybrid_clock.cc:648] HybridClock initialized: now 1790481774837919 us; error 33 us; skew 500 ppm
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:54.841568 18529 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:02:54.842149 18529 webserver.cc:460] Webserver started at http://127.17.184.195:35935/ using document root <none> and password file <none>
I20260927 04:02:54.842468 18529 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:54.842518 18529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:54.842687 18529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:54.843981 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "b6284dcaf5ff430f9833ac4d037ef93d"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "ca3bfdf42b0dc5e0702c3ffaf1416f0f"
server_key_iv: "15cc7be3e2dba0720d87da0e5da9ceab"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.844375 18529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "b6284dcaf5ff430f9833ac4d037ef93d"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "ca3bfdf42b0dc5e0702c3ffaf1416f0f"
server_key_iv: "15cc7be3e2dba0720d87da0e5da9ceab"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.845862 18529 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:02:54.846669 18544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.846860 18529 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:54.846995 18529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "b6284dcaf5ff430f9833ac4d037ef93d"
format_stamp: "Formatted at 2026-09-27 04:02:54 on dist-test-slave-zqrn"
server_key: "ca3bfdf42b0dc5e0702c3ffaf1416f0f"
server_key_iv: "15cc7be3e2dba0720d87da0e5da9ceab"
server_key_version: "encryptionkey@0"
I20260927 04:02:54.847078 18529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:54.859787 18529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:54.862350 18529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:54.862520 18529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:54.862779 18529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:54.863147 18529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:54.863202 18529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.863252 18529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:54.863281 18529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:54.869426 18529 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:38363
I20260927 04:02:54.869495 18657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:38363 every 8 connection(s)
I20260927 04:02:54.870018 18529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:54.877879 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 18529
I20260927 04:02:54.877987 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:54.878165 18147 external_mini_cluster.cc:1428] Setting key e011d7de0127efca5a0615d0db6b4525
I20260927 04:02:54.880169 18256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.870768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:55885 (local address 127.17.184.254:42427)
0927 04:02:54.870877 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:54.870880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:54.871088 (+ 208us) server_negotiation.cc:407] Connection header received
0927 04:02:54.871638 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:54.871640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:54.871652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:54.871711 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:54.872695 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.873213 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.874237 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.874455 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.874542 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.876935 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:54.876936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:54.876937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:54.876944 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:54.878418 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.878957 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.878958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.878959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.879031 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.879355 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.879356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.879357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.879508 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:54.879617 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:54.879996 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 04:02:54.880078 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 04:02:54.880391 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.870739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42427 (local address 127.17.184.195:55885)
0927 04:02:54.870947 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:54.871000 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:54.871523 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:54.871785 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:54.871790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:54.872386 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:54.872586 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:54.872590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.873310 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.873311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:54.874130 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:54.874132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.874554 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.874555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:54.874689 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.875257 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:54.875266 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:54.876845 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:54.878488 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:54.878494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:54.878497 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:54.878836 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:54.879109 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:54.879110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:54.879111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:54.879243 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:54.879624 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:54.879630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:54.879850 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 04:02:54.879957 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 04:02:54.880126 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260927 04:02:54.880781 18658 heartbeater.cc:347] Connected to a master server at 127.17.184.254:42427
I20260927 04:02:54.880894 18658 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:54.881084 18658 heartbeater.cc:511] Master 127.17.184.254:42427 requested a full tablet report, sending...
I20260927 04:02:54.881404 18199 ts_manager.cc:194] Registered new tserver with Master: b6284dcaf5ff430f9833ac4d037ef93d (127.17.184.195:38363)
I20260927 04:02:54.881788 18199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:55885
I20260927 04:02:54.891038 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:02:54.915405 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.906852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.17.184.193:41055)
0927 04:02:54.907071 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:54.907076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:54.907086 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:54.907114 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:54.907116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:54.907129 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:54.907233 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:54.907935 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.908668 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.909406 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.909730 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.909823 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.912199 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:54.912209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:54.912209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:54.912218 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:54.914089 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.914486 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.914490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.914490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.914559 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.914764 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.914765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.914765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.914970 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:54.915107 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:54.915189 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 04:02:54.915277 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":129,"threads_started":1}
W20260927 04:02:54.915789 18304 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53484
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:02:54 dist-test-slave-zqrn krb5kdc[18154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:02:54.939002 18665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:54.931710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53492 (local address 127.17.184.193:41055)
0927 04:02:54.931779 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:54.931782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:54.931895 (+ 113us) server_negotiation.cc:407] Connection header received
0927 04:02:54.931942 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:54.931943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:54.931955 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:54.932006 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:54.932549 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.933045 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.933747 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:54.933941 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:54.933991 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:54.936249 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:54.936250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:54.936250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:54.936264 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:54.937453 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.937843 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.937844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.937844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.937903 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:54.938117 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:54.938118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:54.938119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:54.938222 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:54.938334 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:54.938695 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 04:02:54.938789 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:02:54.939678 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 18261
I20260927 04:02:54.944319 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 18395
I20260927 04:02:54.948702 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 18529
I20260927 04:02:54.952870 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 18167
2026-09-27T04:02:54Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1923 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:55 dist-test-slave-zqrn krb5kdc[18672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:55 dist-test-slave-zqrn krb5kdc[18672](info): set up 2 sockets
Sep 27 04:02:55 dist-test-slave-zqrn krb5kdc[18672](info): commencing operation
krb5kdc: starting...
W20260927 04:02:56.027364 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:02:56.157352 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33893
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:33893
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:56.243638 18685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:56.243860 18685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:56.243882 18685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:56.245092 18685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:56.245134 18685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:56.245148 18685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:56.245162 18685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:56.245174 18685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:56.246675 18685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:33893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:56.246958 18685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:56.247224 18685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:56.252420 18690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:56.252415 18693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:56.252415 18691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:56.252450 18685 server_base.cc:1034] running on GCE node
I20260927 04:02:56.252787 18685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:56.253098 18685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:56.254297 18685 hybrid_clock.cc:648] HybridClock initialized: now 1790481776254283 us; error 35 us; skew 500 ppm
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:56.256940 18685 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:02:56.257373 18685 webserver.cc:460] Webserver started at http://127.17.184.254:44085/ using document root <none> and password file <none>
I20260927 04:02:56.257606 18685 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:56.257679 18685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:56.257827 18685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:56.258876 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "5924be97cda6428e8ead8e0537e9d6cf"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "b2538f2ee7cb031ceeb42754bc934acd"
server_key_iv: "0fc980b1fa17044dc0b048c1969745bd"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.259305 18685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "5924be97cda6428e8ead8e0537e9d6cf"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "b2538f2ee7cb031ceeb42754bc934acd"
server_key_iv: "0fc980b1fa17044dc0b048c1969745bd"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.260625 18685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:56.261418 18700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.261596 18685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:02:56.261695 18685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "5924be97cda6428e8ead8e0537e9d6cf"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "b2538f2ee7cb031ceeb42754bc934acd"
server_key_iv: "0fc980b1fa17044dc0b048c1969745bd"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.261838 18685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:56.273586 18685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:56.275561 18685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:56.275720 18685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:56.279708 18685 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:33893
I20260927 04:02:56.279744 18752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:33893 every 8 connection(s)
I20260927 04:02:56.280092 18685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:56.280920 18753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:56.282258 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 18685
I20260927 04:02:56.282343 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:02:56.282500 18147 external_mini_cluster.cc:1428] Setting key 9879a504cde12936c49e0d7e96b960e7
I20260927 04:02:56.283548 18753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf: Bootstrap starting.
I20260927 04:02:56.284094 18753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:56.284372 18753 log.cc:824] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:56.285002 18753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf: No bootstrap required, opened a new log
I20260927 04:02:56.286710 18753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5924be97cda6428e8ead8e0537e9d6cf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33893 } }
I20260927 04:02:56.286860 18753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:56.286904 18753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5924be97cda6428e8ead8e0537e9d6cf, State: Initialized, Role: FOLLOWER
I20260927 04:02:56.287070 18753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5924be97cda6428e8ead8e0537e9d6cf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33893 } }
I20260927 04:02:56.287164 18753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:56.287195 18753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:56.287238 18753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:56.287760 18753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5924be97cda6428e8ead8e0537e9d6cf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33893 } }
I20260927 04:02:56.287847 18753 leader_election.cc:302] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5924be97cda6428e8ead8e0537e9d6cf; no voters:
I20260927 04:02:56.288020 18753 leader_election.cc:288] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:56.288062 18758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:56.288205 18758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [term 1 LEADER]: Becoming Leader. State: Replica: 5924be97cda6428e8ead8e0537e9d6cf, State: Running, Role: LEADER
I20260927 04:02:56.288265 18753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:56.288312 18758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5924be97cda6428e8ead8e0537e9d6cf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33893 } }
I20260927 04:02:56.288650 18759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5924be97cda6428e8ead8e0537e9d6cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5924be97cda6428e8ead8e0537e9d6cf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33893 } } }
I20260927 04:02:56.288748 18759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:56.288637 18758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5924be97cda6428e8ead8e0537e9d6cf. Latest consensus state: current_term: 1 leader_uuid: "5924be97cda6428e8ead8e0537e9d6cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5924be97cda6428e8ead8e0537e9d6cf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33893 } } }
I20260927 04:02:56.288950 18758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf [sys.catalog]: This master's current role is: LEADER
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:56.289059 18766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:56.289536 18766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:56.292119 18766 catalog_manager.cc:1347] Generated new cluster ID: 9c375582cb0849828684b4d1b85176be
I20260927 04:02:56.292168 18766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:02:56.296226 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.283037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.17.184.254:33893)
0927 04:02:56.283217 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.283227 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.283289 (+ 62us) server_negotiation.cc:407] Connection header received
0927 04:02:56.283740 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.283747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.283795 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.283938 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:56.284640 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.285588 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.286337 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.286737 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.286865 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.289617 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:56.289622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:56.289623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:56.289635 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:56.291578 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.291999 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.292002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.292002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.295009 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.295248 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.295250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.295251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.295605 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:56.295793 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.295868 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.296018 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260927 04:02:56.302045 18766 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:56.302546 18766 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:56.306196 18766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5924be97cda6428e8ead8e0537e9d6cf: Generated new TSK 0
I20260927 04:02:56.306386 18766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:02:56.338480 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33893
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:56.425137 18779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:56.425412 18779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:56.425459 18779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:56.426694 18779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:56.426770 18779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:56.426821 18779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:02:56.428422 18779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:56.428712 18779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:56.428956 18779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:56.434278 18784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:56.434321 18787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:56.434469 18779 server_base.cc:1034] running on GCE node
W20260927 04:02:56.434378 18785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:56.435055 18779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:56.435312 18779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:56.436488 18779 hybrid_clock.cc:648] HybridClock initialized: now 1790481776436490 us; error 41 us; skew 500 ppm
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:56.439205 18779 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:02:56.439647 18779 webserver.cc:460] Webserver started at http://127.17.184.193:32971/ using document root <none> and password file <none>
I20260927 04:02:56.439885 18779 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:56.439955 18779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:56.440097 18779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:56.441193 18779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "84042ad9dff044a7b2527a7af8e4c9fb"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "fca2aac99eb584c936b999a0470eb916"
server_key_iv: "51d4640f688e8ea6be680b76d41c81bf"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.441563 18779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "84042ad9dff044a7b2527a7af8e4c9fb"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "fca2aac99eb584c936b999a0470eb916"
server_key_iv: "51d4640f688e8ea6be680b76d41c81bf"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.443065 18779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:02:56.444033 18794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.444397 18779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:56.444509 18779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "84042ad9dff044a7b2527a7af8e4c9fb"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "fca2aac99eb584c936b999a0470eb916"
server_key_iv: "51d4640f688e8ea6be680b76d41c81bf"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.444608 18779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:56.456192 18779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:56.458631 18779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:56.458860 18779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:56.459291 18779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:56.459630 18779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:56.459687 18779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.459733 18779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:56.459781 18779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.465879 18779 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:39997
I20260927 04:02:56.465983 18907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:39997 every 8 connection(s)
I20260927 04:02:56.466322 18779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:56.474269 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 18779
I20260927 04:02:56.474354 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:56.474542 18147 external_mini_cluster.cc:1428] Setting key d68880e3b49faee31c93b38a6d24933c
I20260927 04:02:56.477064 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.467227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:38465 (local address 127.17.184.254:33893)
0927 04:02:56.467298 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.467301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.467527 (+ 226us) server_negotiation.cc:407] Connection header received
0927 04:02:56.468522 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.468523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.468536 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.468598 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:56.469298 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.469816 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.471003 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.471364 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.471435 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.474025 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:56.474026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:56.474026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:56.474033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:56.475629 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.476085 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.476086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.476087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.476148 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.476405 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.476406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.476407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.476521 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:56.476594 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.476882 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.476968 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 04:02:56.477386 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.467215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33893 (local address 127.17.184.193:38465)
0927 04:02:56.467393 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:56.467432 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:56.468395 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:56.468670 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:56.468674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:56.468954 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:56.469175 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.469179 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.469929 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.469930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.470850 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.470853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.471510 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.471512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.471678 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.472262 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:56.472271 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:56.473926 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:56.475764 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:56.475771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:56.475773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:56.476005 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:56.476217 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:56.476218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:56.476218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:56.476336 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:56.476597 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:56.476604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:56.476749 (+ 145us) client_negotiation.cc:769] Sending connection context
0927 04:02:56.476862 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 04:02:56.477029 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260927 04:02:56.478135 18908 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33893
I20260927 04:02:56.478255 18908 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:56.478458 18908 heartbeater.cc:511] Master 127.17.184.254:33893 requested a full tablet report, sending...
I20260927 04:02:56.479082 18717 ts_manager.cc:194] Registered new tserver with Master: 84042ad9dff044a7b2527a7af8e4c9fb (127.17.184.193:39997)
I20260927 04:02:56.480239 18717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:38465
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:02:56.503244 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33893
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:56.585446 18913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:56.585706 18913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:56.585747 18913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:56.586946 18913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:56.587020 18913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:56.587067 18913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:02:56.588696 18913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:56.588933 18913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:56.589130 18913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:56.594316 18918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:56.594408 18913 server_base.cc:1034] running on GCE node
W20260927 04:02:56.594318 18919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:56.594316 18921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:56.594835 18913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:56.595129 18913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:56.596329 18913 hybrid_clock.cc:648] HybridClock initialized: now 1790481776596308 us; error 40 us; skew 500 ppm
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:56.599103 18913 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:02:56.599591 18913 webserver.cc:460] Webserver started at http://127.17.184.194:44541/ using document root <none> and password file <none>
I20260927 04:02:56.599848 18913 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:56.599898 18913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:56.600036 18913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:56.601106 18913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "248a086c58c0482aa0468b1a3aac3716"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "a61741a06c86ad9fb81532c2ca6c163d"
server_key_iv: "8779f5c2e8a145441223eca29db25b41"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.601485 18913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "248a086c58c0482aa0468b1a3aac3716"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "a61741a06c86ad9fb81532c2ca6c163d"
server_key_iv: "8779f5c2e8a145441223eca29db25b41"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.603070 18913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 04:02:56.604027 18928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.604296 18913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:56.604423 18913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "248a086c58c0482aa0468b1a3aac3716"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "a61741a06c86ad9fb81532c2ca6c163d"
server_key_iv: "8779f5c2e8a145441223eca29db25b41"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.604521 18913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:56.619874 18913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:56.622068 18913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:56.622262 18913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:56.622522 18913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:56.622853 18913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:56.622943 18913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.623004 18913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:56.623035 18913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.629974 18913 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:40305
I20260927 04:02:56.630015 19041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:40305 every 8 connection(s)
I20260927 04:02:56.630357 18913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:56.638494 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 18913
I20260927 04:02:56.638572 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:56.638851 18147 external_mini_cluster.cc:1428] Setting key 8c3d6b8a46ac87b5923f18e8e0463c17
I20260927 04:02:56.640372 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.631209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:56615 (local address 127.17.184.254:33893)
0927 04:02:56.631259 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.631262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.631489 (+ 227us) server_negotiation.cc:407] Connection header received
0927 04:02:56.632472 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.632473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.632485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.632543 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:56.633195 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.633702 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.634758 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.635047 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.635132 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.637317 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:56.637318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:56.637319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:56.637326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:56.638646 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.639180 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.639181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.639182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.639246 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.639537 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.639538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.639538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.639689 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:56.639867 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.640184 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.640274 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:02:56.640630 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.631177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33893 (local address 127.17.184.194:56615)
0927 04:02:56.631370 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:56.631406 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:56.632357 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:56.632618 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:56.632622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:56.632890 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:56.633099 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.633103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.633811 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.633812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.634658 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.634660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.635140 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.635141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.635286 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.635798 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:56.635806 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:56.637204 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:56.638741 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:56.638749 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:56.638751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:56.639090 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:56.639330 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:56.639331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:56.639331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:56.639445 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:56.639847 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:56.639855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:56.640033 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 04:02:56.640159 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 04:02:56.640367 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260927 04:02:56.641079 19042 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33893
I20260927 04:02:56.641199 19042 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:56.641395 19042 heartbeater.cc:511] Master 127.17.184.254:33893 requested a full tablet report, sending...
I20260927 04:02:56.641811 18717 ts_manager.cc:194] Registered new tserver with Master: 248a086c58c0482aa0468b1a3aac3716 (127.17.184.194:40305)
I20260927 04:02:56.642200 18717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:56615
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:02:56.667375 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33893
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:56.750129 19047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:56.750350 19047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:56.750376 19047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:56.751678 19047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:56.751744 19047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:56.751776 19047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:02:56.753341 19047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33893
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:56.753598 19047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:56.753865 19047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:56.759507 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:56.759477 19053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:56.759464 19055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:56.759894 19047 server_base.cc:1034] running on GCE node
I20260927 04:02:56.760146 19047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:56.760433 19047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:56.761621 19047 hybrid_clock.cc:648] HybridClock initialized: now 1790481776761598 us; error 29 us; skew 500 ppm
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:56.764348 19047 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:02:56.764933 19047 webserver.cc:460] Webserver started at http://127.17.184.195:44443/ using document root <none> and password file <none>
I20260927 04:02:56.765268 19047 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:56.765338 19047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:56.765486 19047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:56.766605 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "eec18e73d3d01f2e11b703c5ba37eefa"
server_key_iv: "1290969f5eeab8067761d3e40d74845c"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.767053 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "eec18e73d3d01f2e11b703c5ba37eefa"
server_key_iv: "1290969f5eeab8067761d3e40d74845c"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.768500 19047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:56.769415 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.769680 19047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:56.769896 19047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3"
format_stamp: "Formatted at 2026-09-27 04:02:56 on dist-test-slave-zqrn"
server_key: "eec18e73d3d01f2e11b703c5ba37eefa"
server_key_iv: "1290969f5eeab8067761d3e40d74845c"
server_key_version: "encryptionkey@0"
I20260927 04:02:56.769995 19047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:56.780550 19047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:56.782737 19047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:56.782965 19047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:56.783218 19047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:02:56.783561 19047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:56.783627 19047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.783674 19047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:56.783696 19047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:56.790117 19047 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:34825
I20260927 04:02:56.790184 19175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:34825 every 8 connection(s)
I20260927 04:02:56.790580 19047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:02:56.792380 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 19047
I20260927 04:02:56.792510 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:56.792701 18147 external_mini_cluster.cc:1428] Setting key c4eba459f9fa35043b9d29ef901dc4d0
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:56.800521 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.791458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:48319 (local address 127.17.184.254:33893)
0927 04:02:56.791546 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.791550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.791802 (+ 252us) server_negotiation.cc:407] Connection header received
0927 04:02:56.792847 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.792848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.792860 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.792916 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:56.793548 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.794055 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.795115 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.795316 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.795425 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.797647 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:56.797649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:56.797649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:56.797657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:56.799015 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.799460 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.799461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.799462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.799524 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.799780 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.799789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.799789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.799931 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:56.800015 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.800341 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.800422 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 04:02:56.800738 19178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.791452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33893 (local address 127.17.184.195:48319)
0927 04:02:56.791668 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:56.791708 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:56.792706 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:56.792993 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:56.792997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:56.793255 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:56.793454 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.793458 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.794147 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.794148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.795031 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.795033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.795427 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.795428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.795554 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.796099 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:56.796108 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:56.797541 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:56.799104 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:56.799111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:56.799113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:56.799366 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:56.799601 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:56.799602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:56.799603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:56.799708 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:56.800044 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:56.800050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:56.800183 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 04:02:56.800318 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 04:02:56.800484 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
I20260927 04:02:56.801403 19176 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33893
I20260927 04:02:56.801496 19176 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:56.801707 19176 heartbeater.cc:511] Master 127.17.184.254:33893 requested a full tablet report, sending...
I20260927 04:02:56.802129 18717 ts_manager.cc:194] Registered new tserver with Master: a214fdcdc1b54ce6a428b21a6a60b8f3 (127.17.184.195:34825)
I20260927 04:02:56.802609 18717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:48319
I20260927 04:02:56.804478 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:02:56.812885 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.806188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.17.184.254:33893)
0927 04:02:56.806269 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.806272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.806408 (+ 136us) server_negotiation.cc:407] Connection header received
0927 04:02:56.806455 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.806456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.806465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.806512 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:56.807313 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.807912 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.808733 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.808919 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.808985 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.810111 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:56.810112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:56.810112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:56.810119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:56.811781 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.812107 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.812108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.812108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.812186 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.812394 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.812395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.812396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.812560 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:56.812661 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.812748 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.812814 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:02:56.815122 18717 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:56.816691 18717 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:56.823761 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.819426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.17.184.194:40305)
0927 04:02:56.819859 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.819866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.819883 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:02:56.819916 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.819918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.819931 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.820185 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:56.820268 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.821505 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.822731 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.823403 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.823490 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.823546 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.823635 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.823673 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":100,"threads_started":1}
I20260927 04:02:56.823773 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.819231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:40305 (local address 127.0.0.1:51970)
0927 04:02:56.819421 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:56.819462 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:56.819562 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:56.820063 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:56.820064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:56.820066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:56.820105 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.820106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.821658 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.821659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.822636 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.822639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.823492 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.823493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.823565 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.823570 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:02:56.823624 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 04:02:56.823668 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":48,"threads_started":1}
I20260927 04:02:56.823868 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.819641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.17.184.195:34825)
0927 04:02:56.819817 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.819821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.819831 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:56.819867 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.819868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.819885 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.819966 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:56.820321 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.821015 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.822474 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.823275 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.823357 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.823415 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.823719 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.823772 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260927 04:02:56.824005 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.819231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34825 (local address 127.0.0.1:43734)
0927 04:02:56.819540 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:56.819552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:56.819610 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:56.820182 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:56.820183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:56.820184 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:56.820218 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.820219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.821121 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.821122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.822371 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.822374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.823387 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.823388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.823484 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.823500 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:02:56.823573 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 04:02:56.823638 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":4,"thread_start_us":114,"threads_started":1}
I20260927 04:02:56.824357 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.819255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:39997 (local address 127.0.0.1:32826)
0927 04:02:56.819696 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:56.819706 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:56.819760 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:56.819995 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:56.819998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:56.820001 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:56.820080 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.820083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.821619 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.821620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.822682 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.822685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.823658 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.823659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.823710 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.823713 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:56.823768 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:02:56.823809 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":1,"thread_start_us":122,"threads_started":1}
I20260927 04:02:56.824653 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.819567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.17.184.193:39997)
0927 04:02:56.819716 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.819721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.819820 (+ 99us) server_negotiation.cc:407] Connection header received
0927 04:02:56.819866 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.819868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.819885 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.819965 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:56.820313 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.821499 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.822780 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.823513 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.823576 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.823627 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.824524 (+ 897us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.824570 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260927 04:02:56.824591 18976 tablet_service.cc:1467] Processing CreateTablet for tablet a3849cc371de4ef5b32cfec396165e95 (DEFAULT_TABLE table=test-table [id=43dcb114847f4486b84d569113eee660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:56.824821 19110 tablet_service.cc:1467] Processing CreateTablet for tablet a3849cc371de4ef5b32cfec396165e95 (DEFAULT_TABLE table=test-table [id=43dcb114847f4486b84d569113eee660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:56.824975 18976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3849cc371de4ef5b32cfec396165e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:56.825183 19110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3849cc371de4ef5b32cfec396165e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:56.825210 18842 tablet_service.cc:1467] Processing CreateTablet for tablet a3849cc371de4ef5b32cfec396165e95 (DEFAULT_TABLE table=test-table [id=43dcb114847f4486b84d569113eee660]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:56.825511 18842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3849cc371de4ef5b32cfec396165e95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:56.828284 19193 tablet_bootstrap.cc:492] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3: Bootstrap starting.
I20260927 04:02:56.828702 19194 tablet_bootstrap.cc:492] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716: Bootstrap starting.
I20260927 04:02:56.828725 19195 tablet_bootstrap.cc:492] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb: Bootstrap starting.
I20260927 04:02:56.829105 19193 tablet_bootstrap.cc:654] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:56.829355 19194 tablet_bootstrap.cc:654] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:56.829355 19195 tablet_bootstrap.cc:654] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:56.829723 19193 log.cc:824] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:56.829977 19194 log.cc:824] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:56.829977 19195 log.cc:824] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:56.830559 19193 tablet_bootstrap.cc:492] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3: No bootstrap required, opened a new log
I20260927 04:02:56.830643 19193 ts_tablet_manager.cc:1397] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:56.830763 19194 tablet_bootstrap.cc:492] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716: No bootstrap required, opened a new log
I20260927 04:02:56.830832 19194 ts_tablet_manager.cc:1397] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:02:56.831388 19195 tablet_bootstrap.cc:492] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb: No bootstrap required, opened a new log
I20260927 04:02:56.831512 19195 ts_tablet_manager.cc:1397] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:02:56.832971 19193 raft_consensus.cc:348] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.833196 19193 raft_consensus.cc:374] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:56.833247 19193 raft_consensus.cc:729] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a214fdcdc1b54ce6a428b21a6a60b8f3, State: Initialized, Role: FOLLOWER
I20260927 04:02:56.833216 19194 raft_consensus.cc:348] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.833423 19194 raft_consensus.cc:374] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:56.833397 19193 consensus_queue.cc:260] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.833469 19194 raft_consensus.cc:729] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248a086c58c0482aa0468b1a3aac3716, State: Initialized, Role: FOLLOWER
I20260927 04:02:56.833613 19194 consensus_queue.cc:260] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.833706 19193 ts_tablet_manager.cc:1428] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:56.833845 19176 heartbeater.cc:503] Master 127.17.184.254:33893 was elected leader, sending a full tablet report...
I20260927 04:02:56.833899 19194 ts_tablet_manager.cc:1428] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:02:56.834019 19042 heartbeater.cc:503] Master 127.17.184.254:33893 was elected leader, sending a full tablet report...
I20260927 04:02:56.834406 19195 raft_consensus.cc:348] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.834599 19195 raft_consensus.cc:374] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:56.834653 19195 raft_consensus.cc:729] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84042ad9dff044a7b2527a7af8e4c9fb, State: Initialized, Role: FOLLOWER
I20260927 04:02:56.834801 19195 consensus_queue.cc:260] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.835139 18908 heartbeater.cc:503] Master 127.17.184.254:33893 was elected leader, sending a full tablet report...
I20260927 04:02:56.835140 19195 ts_tablet_manager.cc:1428] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
W20260927 04:02:56.881063 19043 tablet.cc:2369] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:56.933220 19200 raft_consensus.cc:482] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:56.933355 19200 raft_consensus.cc:504] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.933866 19200 leader_election.cc:288] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a214fdcdc1b54ce6a428b21a6a60b8f3 (127.17.184.195:34825), 84042ad9dff044a7b2527a7af8e4c9fb (127.17.184.193:39997)
I20260927 04:02:56.937532 19044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.934132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34825 (local address 127.17.184.194:60273)
0927 04:02:56.934201 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:56.934214 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:56.934265 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:56.934490 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:56.934491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:56.934493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:56.934572 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.934574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.935389 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.935390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.936423 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.936425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.937222 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.937223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.937315 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.937319 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:56.937389 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:02:56.937430 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 04:02:56.937625 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.934232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:53777 (local address 127.17.184.193:39997)
0927 04:02:56.934302 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.934304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.934475 (+ 171us) server_negotiation.cc:407] Connection header received
0927 04:02:56.934521 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.934522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.934533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.934591 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:56.934886 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.935449 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.936664 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.937245 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.937309 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.937317 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.937512 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.937551 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:02:56.937654 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.934133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:39997 (local address 127.17.184.194:53777)
0927 04:02:56.934384 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:56.934395 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:56.934457 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:56.934668 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:56.934669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:56.934672 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:56.934800 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.934802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.935537 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.935538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.936577 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:56.936580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.937350 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.937351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:56.937427 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.937430 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:56.937504 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 04:02:56.937544 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":24,"thread_start_us":148,"threads_started":1}
I20260927 04:02:56.937650 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.934235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:60273 (local address 127.17.184.195:34825)
0927 04:02:56.934302 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.934304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.934314 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:56.934346 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.934347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.934359 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.934448 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:56.934674 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.935308 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.936520 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.937136 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.937212 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.937222 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.937525 (+ 303us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.937576 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 04:02:56.937973 19130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3849cc371de4ef5b32cfec396165e95" candidate_uuid: "248a086c58c0482aa0468b1a3aac3716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" is_pre_election: true
I20260927 04:02:56.937971 18862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3849cc371de4ef5b32cfec396165e95" candidate_uuid: "248a086c58c0482aa0468b1a3aac3716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" is_pre_election: true
I20260927 04:02:56.938149 18862 raft_consensus.cc:2454] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248a086c58c0482aa0468b1a3aac3716 in term 0.
I20260927 04:02:56.938156 19130 raft_consensus.cc:2454] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248a086c58c0482aa0468b1a3aac3716 in term 0.
I20260927 04:02:56.938418 18931 leader_election.cc:302] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248a086c58c0482aa0468b1a3aac3716, 84042ad9dff044a7b2527a7af8e4c9fb; no voters:
I20260927 04:02:56.938567 19200 raft_consensus.cc:2781] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:56.938622 19200 raft_consensus.cc:482] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:56.938657 19200 raft_consensus.cc:3037] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:56.939411 19200 raft_consensus.cc:504] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.939548 19200 leader_election.cc:288] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [CANDIDATE]: Term 1 election: Requested vote from peers a214fdcdc1b54ce6a428b21a6a60b8f3 (127.17.184.195:34825), 84042ad9dff044a7b2527a7af8e4c9fb (127.17.184.193:39997)
I20260927 04:02:56.939697 19130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3849cc371de4ef5b32cfec396165e95" candidate_uuid: "248a086c58c0482aa0468b1a3aac3716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3"
I20260927 04:02:56.939728 18862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3849cc371de4ef5b32cfec396165e95" candidate_uuid: "248a086c58c0482aa0468b1a3aac3716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84042ad9dff044a7b2527a7af8e4c9fb"
I20260927 04:02:56.939800 18862 raft_consensus.cc:3037] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:56.939824 19130 raft_consensus.cc:3037] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:56.940515 18862 raft_consensus.cc:2454] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248a086c58c0482aa0468b1a3aac3716 in term 1.
I20260927 04:02:56.940680 19130 raft_consensus.cc:2454] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248a086c58c0482aa0468b1a3aac3716 in term 1.
I20260927 04:02:56.940689 18931 leader_election.cc:302] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248a086c58c0482aa0468b1a3aac3716, 84042ad9dff044a7b2527a7af8e4c9fb; no voters:
I20260927 04:02:56.940791 19200 raft_consensus.cc:2781] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:56.940950 19200 raft_consensus.cc:686] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [term 1 LEADER]: Becoming Leader. State: Replica: 248a086c58c0482aa0468b1a3aac3716, State: Running, Role: LEADER
I20260927 04:02:56.941056 19200 consensus_queue.cc:237] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } }
I20260927 04:02:56.941932 18717 catalog_manager.cc:5495] T a3849cc371de4ef5b32cfec396165e95 P 248a086c58c0482aa0468b1a3aac3716 reported cstate change: term changed from 0 to 1, leader changed from <none> to 248a086c58c0482aa0468b1a3aac3716 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "248a086c58c0482aa0468b1a3aac3716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a214fdcdc1b54ce6a428b21a6a60b8f3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "248a086c58c0482aa0468b1a3aac3716" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 40305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84042ad9dff044a7b2527a7af8e4c9fb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39997 } health_report { overall_health: UNKNOWN } } }
W20260927 04:02:56.967121 18909 tablet.cc:2369] T a3849cc371de4ef5b32cfec396165e95 P 84042ad9dff044a7b2527a7af8e4c9fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:02:56.984684 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:56.967815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.17.184.193:39997)
0927 04:02:56.967911 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:56.967913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:56.968049 (+ 136us) server_negotiation.cc:407] Connection header received
0927 04:02:56.968130 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:56.968132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:56.968143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:56.968229 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:56.968879 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.969408 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.970239 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:56.970460 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:56.970557 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:56.981746 (+ 11189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:56.981764 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:56.981764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:56.981781 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:56.983543 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.983911 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.983915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.983915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.983988 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:56.984174 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:56.984174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:56.984175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:56.984343 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:56.984462 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:56.984528 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 04:02:56.984598 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 04:02:56 dist-test-slave-zqrn krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:02:57 dist-test-slave-zqrn krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481776, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:02:57.012868 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:57.003243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.17.184.193:39997)
0927 04:02:57.003328 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:57.003331 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:57.003410 (+ 79us) server_negotiation.cc:407] Connection header received
0927 04:02:57.003513 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:57.003514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:57.003525 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:57.003601 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:57.004275 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:57.004896 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:57.006078 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:57.006397 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:57.006471 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:57.009424 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:57.009427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:57.009427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:57.009486 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:57.011393 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:57.011841 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:57.011842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:57.011843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:57.011910 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:57.012166 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:57.012167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:57.012167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:57.012289 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:57.012422 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:57.012639 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 04:02:57.012736 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 04:02:57 dist-test-slave-zqrn krb5kdc[18672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:02:57 dist-test-slave-zqrn krb5kdc[18672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:02:57.037726 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:57.029435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.17.184.193:39997)
0927 04:02:57.029524 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:57.029526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:57.029536 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:57.029639 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:57.029640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:57.029652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:57.029725 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:57.030395 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:57.031037 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:57.031862 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:57.032045 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:57.032105 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:57.034723 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:57.034725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:57.034725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:57.034735 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:57.036311 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:57.036755 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:57.036756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:57.036757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:57.036873 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:57.037158 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:57.037160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:57.037160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:57.037289 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:57.037386 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:57.037572 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 04:02:57.037650 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 04:02:57.038372 18822 tablet_service.cc:2908] Rejecting scan request for tablet a3849cc371de4ef5b32cfec396165e95: Uninitialized: clean time has not yet been initialized
I20260927 04:02:57.039500 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 18779
W20260927 04:02:57.041244 19177 tablet.cc:2369] T a3849cc371de4ef5b32cfec396165e95 P a214fdcdc1b54ce6a428b21a6a60b8f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:57.044697 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 18913
I20260927 04:02:57.049558 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 19047
I20260927 04:02:57.054061 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 18685
2026-09-27T04:02:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2101 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:57 dist-test-slave-zqrn krb5kdc[19217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:57 dist-test-slave-zqrn krb5kdc[19217](info): set up 2 sockets
Sep 27 04:02:57 dist-test-slave-zqrn krb5kdc[19217](info): commencing operation
krb5kdc: starting...
W20260927 04:02:58.106616 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:02:58.232213 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41697
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:41697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:58.314877 19230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:58.315192 19230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:58.315217 19230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:58.316506 19230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:02:58.316553 19230 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 04:02:58.316566 19230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:58.316579 19230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:58.316592 19230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:02:58.316604 19230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:02:58.318252 19230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:41697
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:58.318534 19230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:58.318782 19230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:58.323774 19238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:58.323778 19236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:58.324162 19230 server_base.cc:1034] running on GCE node
W20260927 04:02:58.323812 19235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:58.324509 19230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:58.324877 19230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:58.326052 19230 hybrid_clock.cc:648] HybridClock initialized: now 1790481778326036 us; error 27 us; skew 500 ppm
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:58.328853 19230 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:02:58.329252 19230 webserver.cc:460] Webserver started at http://127.17.184.254:43755/ using document root <none> and password file <none>
I20260927 04:02:58.329542 19230 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:58.329591 19230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:58.329722 19230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:58.330885 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "bd00f909fc0e4950ae0439495d68053a"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "0e169798017f04218c7d65efed2651b9"
server_key_iv: "6a78562b59f374190e14862e9ea4b2ea"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.331375 19230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "bd00f909fc0e4950ae0439495d68053a"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "0e169798017f04218c7d65efed2651b9"
server_key_iv: "6a78562b59f374190e14862e9ea4b2ea"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.332772 19230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:58.333562 19245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.333777 19230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:58.333909 19230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "bd00f909fc0e4950ae0439495d68053a"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "0e169798017f04218c7d65efed2651b9"
server_key_iv: "6a78562b59f374190e14862e9ea4b2ea"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.334000 19230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:58.362970 19230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:58.365525 19230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:58.365710 19230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:58.370568 19230 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:41697
I20260927 04:02:58.370711 19307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:41697 every 8 connection(s)
I20260927 04:02:58.371049 19230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:02:58.371747 19308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:58.374076 19308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a: Bootstrap starting.
I20260927 04:02:58.374677 19308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:58.375034 19308 log.cc:824] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:58.375808 19308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a: No bootstrap required, opened a new log
I20260927 04:02:58.377911 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 19230
I20260927 04:02:58.378034 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:02:58.377995 19308 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00f909fc0e4950ae0439495d68053a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41697 } }
I20260927 04:02:58.378214 19308 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:58.378225 18147 external_mini_cluster.cc:1428] Setting key 243cbdb22b552e0ba6574fc5c70c7b93
I20260927 04:02:58.378244 19308 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd00f909fc0e4950ae0439495d68053a, State: Initialized, Role: FOLLOWER
I20260927 04:02:58.378422 19308 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00f909fc0e4950ae0439495d68053a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41697 } }
I20260927 04:02:58.378525 19308 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:02:58.378563 19308 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:02:58.378609 19308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:58.379382 19308 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00f909fc0e4950ae0439495d68053a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41697 } }
I20260927 04:02:58.379475 19308 leader_election.cc:302] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd00f909fc0e4950ae0439495d68053a; no voters:
I20260927 04:02:58.379678 19308 leader_election.cc:288] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:02:58.379742 19313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:58.379949 19308 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:02:58.379935 19313 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [term 1 LEADER]: Becoming Leader. State: Replica: bd00f909fc0e4950ae0439495d68053a, State: Running, Role: LEADER
I20260927 04:02:58.380182 19313 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00f909fc0e4950ae0439495d68053a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41697 } }
I20260927 04:02:58.380658 19314 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd00f909fc0e4950ae0439495d68053a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00f909fc0e4950ae0439495d68053a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41697 } } }
I20260927 04:02:58.380824 19314 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [sys.catalog]: This master's current role is: LEADER
I20260927 04:02:58.381738 19315 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd00f909fc0e4950ae0439495d68053a. Latest consensus state: current_term: 1 leader_uuid: "bd00f909fc0e4950ae0439495d68053a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd00f909fc0e4950ae0439495d68053a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41697 } } }
I20260927 04:02:58.381891 19315 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a [sys.catalog]: This master's current role is: LEADER
W20260927 04:02:58.382107 19329 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:02:58.382174 19329 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:02:58.382272 19322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:02:58.382483 19322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:02:58.384335 19322 catalog_manager.cc:1347] Generated new cluster ID: 1760ec9fde6a4503a1ef9ee80f6b5e1f
I20260927 04:02:58.384390 19322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:58.390544 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.378937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.17.184.254:41697)
0927 04:02:58.379188 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.379197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.379225 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:02:58.379323 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.379329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.379382 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.379544 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:58.380362 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.381854 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.382924 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.383440 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.383605 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.386562 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:58.386566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:58.386566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:58.386580 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:58.388646 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.389180 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.389184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.389184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.389319 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.389548 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.389549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.389549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.389864 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:58.390018 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.390191 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.390337 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
I20260927 04:02:58.398306 19322 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:02:58.398964 19322 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:02:58.405857 19322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a: Generated new TSK 0
I20260927 04:02:58.406078 19322 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:02:58.436712 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41697
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:58.519752 19334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:58.520006 19334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:58.520046 19334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:58.521224 19334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:58.521296 19334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:58.521330 19334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:02:58.521373 19334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:02:58.522817 19334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:58.523105 19334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:58.523346 19334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:58.528475 19342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:58.528445 19340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:58.528440 19339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:58.528590 19334 server_base.cc:1034] running on GCE node
I20260927 04:02:58.528935 19334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:58.529206 19334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:58.530380 19334 hybrid_clock.cc:648] HybridClock initialized: now 1790481778530368 us; error 27 us; skew 500 ppm
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:58.533087 19334 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:02:58.533519 19334 webserver.cc:460] Webserver started at http://127.17.184.193:35367/ using document root <none> and password file <none>
I20260927 04:02:58.533764 19334 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:58.533869 19334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:58.534027 19334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:58.535187 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "7d7445c6f62343bf8279c4c4b3401d04"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "651a51819d76b539c2cde7327f4db087"
server_key_iv: "2a2137bb928e09fad976f1c50c4712e3"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.535547 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "7d7445c6f62343bf8279c4c4b3401d04"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "651a51819d76b539c2cde7327f4db087"
server_key_iv: "2a2137bb928e09fad976f1c50c4712e3"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.537096 19334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:58.537962 19349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.538178 19334 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:58.538277 19334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "7d7445c6f62343bf8279c4c4b3401d04"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "651a51819d76b539c2cde7327f4db087"
server_key_iv: "2a2137bb928e09fad976f1c50c4712e3"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.538370 19334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:58.557467 19334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:58.559885 19334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:58.560053 19334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:58.560582 19334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:58.560647 19334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.560696 19334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:58.560726 19334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.567760 19334 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:39881
I20260927 04:02:58.567847 19464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:39881 every 8 connection(s)
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:58.568334 19334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:02:58.572009 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.561185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:49261 (local address 127.17.184.254:41697)
0927 04:02:58.561245 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.561247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.561446 (+ 199us) server_negotiation.cc:407] Connection header received
0927 04:02:58.562083 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.562086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.562106 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.562180 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:58.563204 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.564095 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.565709 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.566083 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.566180 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.568953 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:58.568955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:58.568955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:58.568967 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:58.570479 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.570925 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.570927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.570927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.570988 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.571255 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.571257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.571257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.571381 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:58.571459 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.571824 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.571917 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:02:58.572180 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.561155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41697 (local address 127.17.184.193:49261)
0927 04:02:58.561325 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.561361 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.561921 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.562327 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.562332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.562703 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:58.563088 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.563095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.564227 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.564228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.565608 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.565611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.566179 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.566179 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.566284 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.566935 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:58.566946 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:58.568821 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:58.570586 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.570593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.570596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.570830 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.571065 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.571066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.571067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.571184 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.571451 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:58.571456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:58.571644 (+ 188us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.571776 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.571948 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260927 04:02:58.572386 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 19334
I20260927 04:02:58.572494 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:02:58.572680 18147 external_mini_cluster.cc:1428] Setting key 4f307babb75c9f13e8e7cd1855679aad
I20260927 04:02:58.572746 19465 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41697
I20260927 04:02:58.572875 19465 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:58.573081 19465 heartbeater.cc:511] Master 127.17.184.254:41697 requested a full tablet report, sending...
I20260927 04:02:58.573645 19261 ts_manager.cc:194] Registered new tserver with Master: 7d7445c6f62343bf8279c4c4b3401d04 (127.17.184.193:39881)
I20260927 04:02:58.574565 19261 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:49261
I20260927 04:02:58.581285 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.574988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:42313 (local address 127.17.184.254:41697)
0927 04:02:58.575026 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.575028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.575164 (+ 136us) server_negotiation.cc:407] Connection header received
0927 04:02:58.575221 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.575222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.575232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.575292 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:58.575680 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.576212 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.577288 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.577491 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.577537 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.578560 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:58.578561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:58.578561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:58.578567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:58.579967 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.580387 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.580388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.580389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.580457 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.580773 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.580774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.580774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.580882 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:58.580941 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.581145 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.581222 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:02:58.581350 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.574945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41697 (local address 127.17.184.193:42313)
0927 04:02:58.575084 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.575097 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.575160 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.575297 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.575298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.575521 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:58.575601 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.575602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.576308 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.576309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.577198 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.577200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.577545 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.577546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.577636 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.578024 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:58.578029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:58.578466 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:58.580030 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.580031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.580031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.580274 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.580582 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.580582 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.580583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.580652 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.580956 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:58.580957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:58.581008 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.581110 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.581222 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:02:58.602433 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41697
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:58.685226 19474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:58.685439 19474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:58.685464 19474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:58.686661 19474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:58.686712 19474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:58.686733 19474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:02:58.686779 19474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:02:58.688298 19474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:58.688561 19474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:58.688822 19474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:58.694154 19480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:58.694234 19479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:58.694169 19482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:58.694303 19474 server_base.cc:1034] running on GCE node
I20260927 04:02:58.694883 19474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:58.695195 19474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:58.696425 19474 hybrid_clock.cc:648] HybridClock initialized: now 1790481778696391 us; error 51 us; skew 500 ppm
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:58.699132 19474 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:02:58.699679 19474 webserver.cc:460] Webserver started at http://127.17.184.194:37537/ using document root <none> and password file <none>
I20260927 04:02:58.699934 19474 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:58.700004 19474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:58.700152 19474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:58.701270 19474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "0e0e2c48dff24502a950f3d18d6dc630"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "94b33737bcdbd28f1761acf6771806e8"
server_key_iv: "7fb04ae05945b491c52da28f41a98888"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.701767 19474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "0e0e2c48dff24502a950f3d18d6dc630"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "94b33737bcdbd28f1761acf6771806e8"
server_key_iv: "7fb04ae05945b491c52da28f41a98888"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.703729 19474 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 04:02:58.704555 19489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.704785 19474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:58.704869 19474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "0e0e2c48dff24502a950f3d18d6dc630"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "94b33737bcdbd28f1761acf6771806e8"
server_key_iv: "7fb04ae05945b491c52da28f41a98888"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.704968 19474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:58.715566 19474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:58.717583 19474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:58.717746 19474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:58.718309 19474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:58.718348 19474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.718371 19474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:58.718384 19474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:58.725592 19474 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:37991
I20260927 04:02:58.725672 19604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:37991 every 8 connection(s)
I20260927 04:02:58.725967 19474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:02:58.727224 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 19474
I20260927 04:02:58.727329 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:02:58.727530 18147 external_mini_cluster.cc:1428] Setting key be991d1d96f1f8a53d4b86dc5d322cc2
I20260927 04:02:58.728924 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.718716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:60119 (local address 127.17.184.254:41697)
0927 04:02:58.718799 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.718801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.719069 (+ 268us) server_negotiation.cc:407] Connection header received
0927 04:02:58.719589 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.719591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.719605 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.719687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:58.720465 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.721311 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.722358 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.722712 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.722806 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.725611 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:58.725614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:58.725614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:58.725623 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:58.727394 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.727897 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.727898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.727898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.727963 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.728213 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.728214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.728215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.728352 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:58.728431 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.728741 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.728827 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:02:58.729171 19498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.718697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41697 (local address 127.17.184.194:60119)
0927 04:02:58.718919 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.718962 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.719487 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.719803 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.719807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.720098 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:58.720367 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.720374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.721408 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.721409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.722251 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.722253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.722834 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.722836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.723006 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.723529 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:58.723537 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:58.725507 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:58.727481 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.727487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.727488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.727823 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.728031 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.728032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.728032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.728132 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.728431 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:58.728439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:58.728604 (+ 165us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.728718 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.728944 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":135,"threads_started":1}
I20260927 04:02:58.730497 19605 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41697
I20260927 04:02:58.730559 19605 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:58.730775 19605 heartbeater.cc:511] Master 127.17.184.254:41697 requested a full tablet report, sending...
I20260927 04:02:58.731173 19262 ts_manager.cc:194] Registered new tserver with Master: 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991)
I20260927 04:02:58.731712 19262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:60119
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
I20260927 04:02:58.737380 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.730665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41697 (local address 127.17.184.194:50295)
0927 04:02:58.730800 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.730811 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.730889 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.731102 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.731103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.731293 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:58.731343 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.731344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.732082 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.732083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.733105 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.733108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.733488 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.733489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.733579 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.734072 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:58.734078 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:58.734507 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:58.736226 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.736227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.736228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.736464 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.736665 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.736666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.736666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.736741 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.737070 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:58.737070 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:58.737147 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.737225 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.737309 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":26,"thread_start_us":80,"threads_started":1}
I20260927 04:02:58.737565 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.730676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:50295 (local address 127.17.184.254:41697)
0927 04:02:58.730720 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.730722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.730865 (+ 143us) server_negotiation.cc:407] Connection header received
0927 04:02:58.730991 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.730992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.731006 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.731035 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:58.731427 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.731980 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.733217 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.733402 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.733466 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.734600 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:58.734601 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:58.734601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:58.734608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:58.736138 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.736539 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.736540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.736541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.736593 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.736817 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.736817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.736817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.736986 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:58.737054 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.737297 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.737412 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:02:58.757627 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41697
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:02:58.843278 19614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:02:58.843492 19614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:02:58.843518 19614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:02:58.844748 19614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:02:58.844792 19614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:02:58.844813 19614 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:02:58.844831 19614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:02:58.846330 19614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:02:58.846580 19614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:02:58.846822 19614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:02:58.852109 19619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:02:58.852094 19622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:58.852368 19614 server_base.cc:1034] running on GCE node
W20260927 04:02:58.852093 19620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:02:58.852676 19614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:02:58.852952 19614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:02:58.854130 19614 hybrid_clock.cc:648] HybridClock initialized: now 1790481778854117 us; error 32 us; skew 500 ppm
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:02:58.856994 19614 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:02:58.857477 19614 webserver.cc:460] Webserver started at http://127.17.184.195:36771/ using document root <none> and password file <none>
I20260927 04:02:58.857674 19614 fs_manager.cc:359] Metadata directory not provided
I20260927 04:02:58.857715 19614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:02:58.857808 19614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:02:58.858809 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "d33d45be0bbc45c1abef3db02708cdad"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "4b32e73c3628013658846a8f31af21e6"
server_key_iv: "bd8bf1729d5882ab480ae389cd7b21d4"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.859182 19614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "d33d45be0bbc45c1abef3db02708cdad"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "4b32e73c3628013658846a8f31af21e6"
server_key_iv: "bd8bf1729d5882ab480ae389cd7b21d4"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.860436 19614 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:58.861123 19629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.861295 19614 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:58.861364 19614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "d33d45be0bbc45c1abef3db02708cdad"
format_stamp: "Formatted at 2026-09-27 04:02:58 on dist-test-slave-zqrn"
server_key: "4b32e73c3628013658846a8f31af21e6"
server_key_iv: "bd8bf1729d5882ab480ae389cd7b21d4"
server_key_version: "encryptionkey@0"
I20260927 04:02:58.861413 19614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:02:58.869508 19614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:02:58.871558 19614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:02:58.871666 19614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:02:58.872212 19614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:02:58.872277 19614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.872318 19614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:02:58.872359 19614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:58.878743 19614 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:44703
I20260927 04:02:58.878820 19744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:44703 every 8 connection(s)
I20260927 04:02:58.879144 19614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:58.882719 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 19614
I20260927 04:02:58.882841 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:02:58.883065 18147 external_mini_cluster.cc:1428] Setting key 6118cd161c022b1c72ae40a51b850bcc
I20260927 04:02:58.883520 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.872734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41697 (local address 127.17.184.195:49563)
0927 04:02:58.872953 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.873007 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.873508 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.873774 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.873779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.874221 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:58.874506 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.874512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.875664 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.875664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.877007 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.877010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.877476 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.877478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.877622 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.878270 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:58.878280 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:58.880079 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:58.882037 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.882043 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.882045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.882277 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.882496 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.882497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.882497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.882632 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.882928 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:58.882935 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:58.883081 (+ 146us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.883210 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.883353 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":141,"threads_started":1}
I20260927 04:02:58.883850 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.872884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:49563 (local address 127.17.184.254:41697)
0927 04:02:58.872948 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.872950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.873093 (+ 143us) server_negotiation.cc:407] Connection header received
0927 04:02:58.873626 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.873628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.873643 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.873700 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:58.874649 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.875539 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.877097 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.877397 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.877469 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.880189 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:58.880191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:58.880191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:58.880200 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:58.881939 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.882355 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.882356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.882357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.882416 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.882707 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.882708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.882709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.882854 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:58.882939 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.883672 (+ 733us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.883762 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:02:58.884249 19745 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41697
I20260927 04:02:58.884356 19745 heartbeater.cc:464] Registering TS with master...
I20260927 04:02:58.884532 19745 heartbeater.cc:511] Master 127.17.184.254:41697 requested a full tablet report, sending...
I20260927 04:02:58.885002 19262 ts_manager.cc:194] Registered new tserver with Master: d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703)
I20260927 04:02:58.885404 19262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:49563
I20260927 04:02:58.891119 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.885282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41697 (local address 127.17.184.195:37415)
0927 04:02:58.885424 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.885435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.885571 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.885751 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.885752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.885908 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:58.885960 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.885961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.886661 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.886662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.887273 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.887275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.887604 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.887605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.887691 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.888081 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:58.888085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:58.888371 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:58.889953 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.889954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.889954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.890115 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.890355 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:58.890355 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:58.890355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:58.890450 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:58.890765 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:58.890766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:58.890838 (+ 72us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.890954 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.891048 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260927 04:02:58.891130 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.885320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:37415 (local address 127.17.184.254:41697)
0927 04:02:58.885372 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.885373 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.885489 (+ 116us) server_negotiation.cc:407] Connection header received
0927 04:02:58.885646 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.885647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.885657 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.885712 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:58.886040 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.886539 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.887351 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.887532 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.887595 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.888448 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:58.888449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:58.888449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:58.888455 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:58.889873 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.890204 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.890205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.890205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.890283 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.890537 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.890538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.890538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.890696 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:58.890758 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.891003 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.891069 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:02:58.894636 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:02:58 dist-test-slave-zqrn krb5kdc[19217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:58.917520 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.909962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.17.184.254:41697)
0927 04:02:58.910027 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.910029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.910039 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:58.910069 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.910070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.910085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.910136 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:58.910720 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.911357 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.912094 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.912285 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.912359 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.914760 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:58.914761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:58.914761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:58.914769 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:58.916397 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.916755 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.916756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.916757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.916816 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:58.917002 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:58.917002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:58.917003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:58.917108 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:58.917183 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.917339 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.917405 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:02:58.919239 19262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:58.920884 19262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:58.927788 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.923756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40564 (local address 127.17.184.195:44703)
0927 04:02:58.923944 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.923949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.923959 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:58.923998 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.924000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.924015 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.924254 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:58.924317 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.925429 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.926728 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.927342 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.927439 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.927500 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.927629 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.927673 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":86,"threads_started":1}
I20260927 04:02:58.927781 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.923359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44703 (local address 127.0.0.1:40564)
0927 04:02:58.923546 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.923587 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.923658 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.924116 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.924117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.924119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:58.924179 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.924181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.925520 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.925521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.926630 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.926632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.927440 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.927441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.927520 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.927533 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.927609 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.927683 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":51,"thread_start_us":89,"threads_started":1}
I20260927 04:02:58.927784 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.923360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:37991 (local address 127.0.0.1:39130)
0927 04:02:58.923615 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.923623 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.923664 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.924062 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.924065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.924068 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:58.924147 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.924149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.925170 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.925172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.926477 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.926480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.927483 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.927484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.927563 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.927567 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.927598 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.927683 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260927 04:02:58.928000 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.923547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.17.184.194:37991)
0927 04:02:58.923838 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.923845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.923856 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:02:58.923898 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.923900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.923915 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.924233 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:58.924309 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.925098 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.926587 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.927382 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.927737 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.927813 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.927836 (+ 23us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.927879 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
I20260927 04:02:58.928176 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.923415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:39881 (local address 127.0.0.1:50026)
0927 04:02:58.923770 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.923778 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.923814 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.924160 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.924161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.924163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:58.924191 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.924192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.925542 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.925543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.926483 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.926486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.927540 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.927540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.927598 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.927602 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.927649 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.927686 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":1,"thread_start_us":66,"threads_started":1}
I20260927 04:02:58.928478 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.923645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.17.184.193:39881)
0927 04:02:58.923944 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.923949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.923959 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:02:58.923998 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.924000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.924015 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.924093 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:58.924267 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.925418 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.926614 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.927382 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.927458 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.927512 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.927831 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.928368 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":76,"threads_started":1}
I20260927 04:02:58.928892 19539 tablet_service.cc:1467] Processing CreateTablet for tablet bba8d0b752714d7b87eac25ac379dc2e (DEFAULT_TABLE table=test-table [id=dd2492f0280b42a680aa9912a568ae8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:58.929126 19399 tablet_service.cc:1467] Processing CreateTablet for tablet bba8d0b752714d7b87eac25ac379dc2e (DEFAULT_TABLE table=test-table [id=dd2492f0280b42a680aa9912a568ae8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:58.929292 19539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba8d0b752714d7b87eac25ac379dc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:58.929193 19662 tablet_service.cc:1467] Processing CreateTablet for tablet bba8d0b752714d7b87eac25ac379dc2e (DEFAULT_TABLE table=test-table [id=dd2492f0280b42a680aa9912a568ae8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:58.929447 19399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba8d0b752714d7b87eac25ac379dc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:58.929589 19662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bba8d0b752714d7b87eac25ac379dc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:58.932420 19768 tablet_bootstrap.cc:492] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: Bootstrap starting.
I20260927 04:02:58.933108 19770 tablet_bootstrap.cc:492] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: Bootstrap starting.
I20260927 04:02:58.933457 19768 tablet_bootstrap.cc:654] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:58.933781 19770 tablet_bootstrap.cc:654] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:58.933851 19768 log.cc:824] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:58.934144 19770 log.cc:824] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:58.934474 19769 tablet_bootstrap.cc:492] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Bootstrap starting.
I20260927 04:02:58.934679 19768 tablet_bootstrap.cc:492] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: No bootstrap required, opened a new log
I20260927 04:02:58.934763 19768 ts_tablet_manager.cc:1397] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:02:58.934897 19770 tablet_bootstrap.cc:492] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: No bootstrap required, opened a new log
I20260927 04:02:58.935098 19770 ts_tablet_manager.cc:1397] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 04:02:58.935218 19769 tablet_bootstrap.cc:654] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:58.935536 19769 log.cc:824] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Log is configured to *not* fsync() on all Append() calls
I20260927 04:02:58.936800 19768 raft_consensus.cc:348] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:58.936970 19768 raft_consensus.cc:374] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:58.937009 19768 raft_consensus.cc:729] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33d45be0bbc45c1abef3db02708cdad, State: Initialized, Role: FOLLOWER
I20260927 04:02:58.936933 19770 raft_consensus.cc:348] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:58.937089 19770 raft_consensus.cc:374] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:58.937134 19770 raft_consensus.cc:729] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7445c6f62343bf8279c4c4b3401d04, State: Initialized, Role: FOLLOWER
I20260927 04:02:58.937134 19768 consensus_queue.cc:260] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:58.937264 19770 consensus_queue.cc:260] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:58.937448 19768 ts_tablet_manager.cc:1428] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:02:58.937517 19770 ts_tablet_manager.cc:1428] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 04:02:58.937568 19465 heartbeater.cc:503] Master 127.17.184.254:41697 was elected leader, sending a full tablet report...
I20260927 04:02:58.937598 19745 heartbeater.cc:503] Master 127.17.184.254:41697 was elected leader, sending a full tablet report...
I20260927 04:02:58.938125 19769 tablet_bootstrap.cc:492] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: No bootstrap required, opened a new log
I20260927 04:02:58.938195 19769 ts_tablet_manager.cc:1397] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 04:02:58.939921 19769 raft_consensus.cc:348] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:58.940085 19769 raft_consensus.cc:374] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:58.940131 19769 raft_consensus.cc:729] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0e2c48dff24502a950f3d18d6dc630, State: Initialized, Role: FOLLOWER
I20260927 04:02:58.940258 19769 consensus_queue.cc:260] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:58.940529 19769 ts_tablet_manager.cc:1428] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 04:02:58.940615 19605 heartbeater.cc:503] Master 127.17.184.254:41697 was elected leader, sending a full tablet report...
W20260927 04:02:58.976586 19606 tablet.cc:2369] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:58.993379 19775 raft_consensus.cc:482] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:58.993500 19775 raft_consensus.cc:504] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:58.993882 19775 leader_election.cc:288] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991), d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703)
I20260927 04:02:58.994439 19774 raft_consensus.cc:482] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:58.994560 19774 raft_consensus.cc:504] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:58.995263 19774 leader_election.cc:288] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991), 7d7445c6f62343bf8279c4c4b3401d04 (127.17.184.193:39881)
I20260927 04:02:58.997851 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.994155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44703 (local address 127.17.184.193:51565)
0927 04:02:58.994234 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.994246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.994285 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.994478 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.994479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.994481 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:58.994554 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.994556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.995699 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.995701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.996890 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.996892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.997589 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.997590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.997673 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.997677 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.997724 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.997769 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260927 04:02:58.998095 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.994210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:51565 (local address 127.17.184.195:44703)
0927 04:02:58.994258 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.994260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.994301 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:02:58.994393 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.994394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.994406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.994469 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:58.994631 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.995464 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.996983 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.997494 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.997554 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.997562 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.998004 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.998039 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:02:58.998350 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba8d0b752714d7b87eac25ac379dc2e" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33d45be0bbc45c1abef3db02708cdad" is_pre_election: true
I20260927 04:02:58.998441 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.994383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:48439 (local address 127.17.184.194:37991)
0927 04:02:58.994430 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.994432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.994489 (+ 57us) server_negotiation.cc:407] Connection header received
0927 04:02:58.994590 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.994592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.994601 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.994632 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:58.994849 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.995554 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.997548 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.998068 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.998109 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.998116 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.998345 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.998377 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:02:58.998448 19699 raft_consensus.cc:2454] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 0.
I20260927 04:02:58.998657 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.995146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:39881 (local address 127.17.184.195:53185)
0927 04:02:58.995222 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.995232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.995268 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.995666 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.995667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.995668 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:58.995725 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.995727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.996588 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.996589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.997706 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.997708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.998404 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.998405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.998471 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.998473 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.998543 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.998588 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 04:02:58.998646 19352 leader_election.cc:302] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7445c6f62343bf8279c4c4b3401d04, d33d45be0bbc45c1abef3db02708cdad; no voters:
I20260927 04:02:58.998739 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.994160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:37991 (local address 127.17.184.193:48439)
0927 04:02:58.994426 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.994436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.994516 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.994702 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.994702 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.994704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:58.994781 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.994782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.995651 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.995652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.997452 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.997455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.998188 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.998188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.998260 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.998263 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.998548 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.998586 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":74,"thread_start_us":112,"threads_started":1}
I20260927 04:02:58.999061 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba8d0b752714d7b87eac25ac379dc2e" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0e2c48dff24502a950f3d18d6dc630" is_pre_election: true
I20260927 04:02:58.999157 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.995237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:53185 (local address 127.17.184.193:39881)
0927 04:02:58.995561 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.995563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.995570 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:02:58.995591 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.995592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.995602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.995651 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:58.995799 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.996500 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.997785 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.998302 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.998368 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.998376 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.999056 (+ 680us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.999104 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301}
I20260927 04:02:58.999207 19559 raft_consensus.cc:2454] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 0.
I20260927 04:02:58.999420 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.995146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:37991 (local address 127.17.184.195:46793)
0927 04:02:58.995471 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:58.995480 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:58.995539 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:58.995871 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:58.995871 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:58.995873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:02:58.995934 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.995935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.996888 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.996889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.997998 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:58.998001 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.998739 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.998740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:58.998836 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.998840 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:02:58.998889 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 04:02:58.999338 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":169,"thread_start_us":109,"threads_started":1}
I20260927 04:02:58.999372 19419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba8d0b752714d7b87eac25ac379dc2e" candidate_uuid: "d33d45be0bbc45c1abef3db02708cdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7445c6f62343bf8279c4c4b3401d04" is_pre_election: true
I20260927 04:02:58.999465 19419 raft_consensus.cc:2454] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33d45be0bbc45c1abef3db02708cdad in term 0.
I20260927 04:02:58.999620 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:58.995580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:46793 (local address 127.17.184.194:37991)
0927 04:02:58.995726 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:58.995728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:58.995735 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:02:58.995753 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:58.995754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:58.995762 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:58.995831 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:02:58.996004 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.996787 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.998085 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:58.998620 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:58.998685 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:58.998691 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:58.999499 (+ 808us) server_negotiation.cc:299] Negotiation successful
0927 04:02:58.999542 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260927 04:02:58.999646 19633 leader_election.cc:302] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7445c6f62343bf8279c4c4b3401d04, d33d45be0bbc45c1abef3db02708cdad; no voters:
I20260927 04:02:58.999662 19775 raft_consensus.cc:2781] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:58.999713 19775 raft_consensus.cc:482] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:58.999738 19775 raft_consensus.cc:3037] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:58.999768 19774 raft_consensus.cc:2781] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:58.999812 19774 raft_consensus.cc:482] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:58.999840 19774 raft_consensus.cc:3037] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:59.000005 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba8d0b752714d7b87eac25ac379dc2e" candidate_uuid: "d33d45be0bbc45c1abef3db02708cdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0e2c48dff24502a950f3d18d6dc630" is_pre_election: true
I20260927 04:02:59.000089 19559 raft_consensus.cc:2454] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33d45be0bbc45c1abef3db02708cdad in term 0.
I20260927 04:02:59.000728 19775 raft_consensus.cc:504] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:59.001001 19775 leader_election.cc:288] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 election: Requested vote from peers 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991), d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703)
I20260927 04:02:59.001020 19774 raft_consensus.cc:504] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:59.001070 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba8d0b752714d7b87eac25ac379dc2e" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0e2c48dff24502a950f3d18d6dc630"
I20260927 04:02:59.001020 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba8d0b752714d7b87eac25ac379dc2e" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33d45be0bbc45c1abef3db02708cdad"
I20260927 04:02:59.001153 19699 raft_consensus.cc:2379] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7d7445c6f62343bf8279c4c4b3401d04 in current term 1: Already voted for candidate d33d45be0bbc45c1abef3db02708cdad in this term.
I20260927 04:02:59.001134 19559 raft_consensus.cc:3037] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:59.001183 19774 leader_election.cc:288] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [CANDIDATE]: Term 1 election: Requested vote from peers 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991), 7d7445c6f62343bf8279c4c4b3401d04 (127.17.184.193:39881)
I20260927 04:02:59.001343 19419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba8d0b752714d7b87eac25ac379dc2e" candidate_uuid: "d33d45be0bbc45c1abef3db02708cdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7445c6f62343bf8279c4c4b3401d04"
I20260927 04:02:59.001433 19419 raft_consensus.cc:2379] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d33d45be0bbc45c1abef3db02708cdad in current term 1: Already voted for candidate 7d7445c6f62343bf8279c4c4b3401d04 in this term.
I20260927 04:02:59.001473 19558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bba8d0b752714d7b87eac25ac379dc2e" candidate_uuid: "d33d45be0bbc45c1abef3db02708cdad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0e2c48dff24502a950f3d18d6dc630"
I20260927 04:02:59.001816 19631 leader_election.cc:302] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d33d45be0bbc45c1abef3db02708cdad; no voters: 0e0e2c48dff24502a950f3d18d6dc630, 7d7445c6f62343bf8279c4c4b3401d04
I20260927 04:02:59.001931 19774 raft_consensus.cc:2726] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 04:02:59.002115 19559 raft_consensus.cc:2454] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 1.
I20260927 04:02:59.002310 19351 leader_election.cc:302] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e0e2c48dff24502a950f3d18d6dc630, 7d7445c6f62343bf8279c4c4b3401d04; no voters: d33d45be0bbc45c1abef3db02708cdad
I20260927 04:02:59.002410 19775 raft_consensus.cc:2781] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:59.002558 19775 raft_consensus.cc:686] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 LEADER]: Becoming Leader. State: Replica: 7d7445c6f62343bf8279c4c4b3401d04, State: Running, Role: LEADER
I20260927 04:02:59.002652 19775 consensus_queue.cc:237] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } }
I20260927 04:02:59.003575 19261 catalog_manager.cc:5495] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d7445c6f62343bf8279c4c4b3401d04 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } health_report { overall_health: HEALTHY } } }
I20260927 04:02:59.023186 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.020659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.17.184.193:39881)
0927 04:02:59.020727 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:59.020729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:59.020737 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:02:59.020811 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:59.020812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:59.020832 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:59.020889 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:59.021013 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.021487 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.022281 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.022493 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.022564 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.022778 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:59.022954 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:59.023040 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:59.023082 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 04:02:59.023106 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 04:02:59.025065 19559 raft_consensus.cc:1261] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Refusing update from remote peer 7d7445c6f62343bf8279c4c4b3401d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:59.025065 19699 raft_consensus.cc:1261] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Refusing update from remote peer 7d7445c6f62343bf8279c4c4b3401d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:59.025353 19780 consensus_queue.cc:1035] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:59.025498 19775 consensus_queue.cc:1035] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:59.027971 19783 mvcc.cc:205] Tried to move back new op lower bound from 7333813366884012032 to 7333813366795431936. Current Snapshot: MvccSnapshot[applied={T|T < 7333813366884012032}]
I20260927 04:02:59.028209 19785 mvcc.cc:205] Tried to move back new op lower bound from 7333813366884012032 to 7333813366795431936. Current Snapshot: MvccSnapshot[applied={T|T < 7333813366884012032}]
I20260927 04:02:59.032075 19262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:59.032163 19262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:59.032982 19262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a: Sending DeleteTablet for 3 replicas of tablet bba8d0b752714d7b87eac25ac379dc2e
I20260927 04:02:59.033320 19539 tablet_service.cc:1514] Processing DeleteTablet for tablet bba8d0b752714d7b87eac25ac379dc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:59 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:39130
I20260927 04:02:59.033403 19662 tablet_service.cc:1514] Processing DeleteTablet for tablet bba8d0b752714d7b87eac25ac379dc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:59 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:40564
I20260927 04:02:59.033448 19399 tablet_service.cc:1514] Processing DeleteTablet for tablet bba8d0b752714d7b87eac25ac379dc2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:59 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:50026
I20260927 04:02:59.033771 19793 tablet_replica.cc:330] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: stopping tablet replica
I20260927 04:02:59.033962 19793 raft_consensus.cc:2229] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:59.034087 19793 raft_consensus.cc:2258] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:59.034037 19262 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:02:59.034194 19262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:59.034578 19793 ts_tablet_manager.cc:1905] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:59.035496 19399 tablet_service.cc:1467] Processing CreateTablet for tablet b33afb6a855c4679a93d2f080694498e (DEFAULT_TABLE table=test-table [id=d7d9d60f0cbc4b3cb82af6d916703c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:59.035501 19539 tablet_service.cc:1467] Processing CreateTablet for tablet b33afb6a855c4679a93d2f080694498e (DEFAULT_TABLE table=test-table [id=d7d9d60f0cbc4b3cb82af6d916703c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:59.035624 19399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33afb6a855c4679a93d2f080694498e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:59.035624 19539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33afb6a855c4679a93d2f080694498e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:59.035780 19794 tablet_replica.cc:330] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: stopping tablet replica
I20260927 04:02:59.035887 19794 raft_consensus.cc:2229] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:59.035991 19794 raft_consensus.cc:2258] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:59.036764 19793 ts_tablet_manager.cc:1918] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:59.036827 19793 log.cc:1196] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/bba8d0b752714d7b87eac25ac379dc2e
I20260927 04:02:59.037012 19769 tablet_bootstrap.cc:492] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: Bootstrap starting.
I20260927 04:02:59.037088 19793 ts_tablet_manager.cc:1939] T bba8d0b752714d7b87eac25ac379dc2e P 0e0e2c48dff24502a950f3d18d6dc630: Deleting consensus metadata
I20260927 04:02:59.037297 19795 tablet_replica.cc:330] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: stopping tablet replica
I20260927 04:02:59.037400 19795 raft_consensus.cc:2229] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:02:59.037415 19769 tablet_bootstrap.cc:654] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:59.037546 19795 raft_consensus.cc:2258] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:59.037639 19662 tablet_service.cc:1467] Processing CreateTablet for tablet b33afb6a855c4679a93d2f080694498e (DEFAULT_TABLE table=test-table [id=d7d9d60f0cbc4b3cb82af6d916703c31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:02:59.037751 19662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33afb6a855c4679a93d2f080694498e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:59.037734 19770 tablet_bootstrap.cc:492] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: Bootstrap starting.
I20260927 04:02:59.038020 19247 catalog_manager.cc:4841] TS 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991): tablet bba8d0b752714d7b87eac25ac379dc2e (table test-table [id=dd2492f0280b42a680aa9912a568ae8b]) successfully deleted
I20260927 04:02:59.038087 19794 ts_tablet_manager.cc:1905] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:59.038158 19770 tablet_bootstrap.cc:654] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:59.038812 19770 tablet_bootstrap.cc:492] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: No bootstrap required, opened a new log
I20260927 04:02:59.038856 19770 ts_tablet_manager.cc:1397] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:59.039033 19770 raft_consensus.cc:348] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.039120 19770 raft_consensus.cc:374] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:59.039143 19770 raft_consensus.cc:729] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7445c6f62343bf8279c4c4b3401d04, State: Initialized, Role: FOLLOWER
I20260927 04:02:59.039191 19770 consensus_queue.cc:260] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.039294 19770 ts_tablet_manager.cc:1428] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:59.039418 19768 tablet_bootstrap.cc:492] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: Bootstrap starting.
I20260927 04:02:59.039783 19768 tablet_bootstrap.cc:654] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:59.039803 19795 ts_tablet_manager.cc:1905] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:59.040414 19768 tablet_bootstrap.cc:492] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: No bootstrap required, opened a new log
I20260927 04:02:59.040462 19768 ts_tablet_manager.cc:1397] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:59.040606 19768 raft_consensus.cc:348] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.040674 19768 raft_consensus.cc:374] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:59.040696 19768 raft_consensus.cc:729] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33d45be0bbc45c1abef3db02708cdad, State: Initialized, Role: FOLLOWER
I20260927 04:02:59.040735 19768 consensus_queue.cc:260] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.040845 19768 ts_tablet_manager.cc:1428] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:59.041714 19794 ts_tablet_manager.cc:1918] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:59.041795 19794 log.cc:1196] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/bba8d0b752714d7b87eac25ac379dc2e
I20260927 04:02:59.042049 19794 ts_tablet_manager.cc:1939] T bba8d0b752714d7b87eac25ac379dc2e P d33d45be0bbc45c1abef3db02708cdad: Deleting consensus metadata
I20260927 04:02:59.042245 19795 ts_tablet_manager.cc:1918] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:02:59.042328 19795 log.cc:1196] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/bba8d0b752714d7b87eac25ac379dc2e
I20260927 04:02:59.042578 19795 ts_tablet_manager.cc:1939] T bba8d0b752714d7b87eac25ac379dc2e P 7d7445c6f62343bf8279c4c4b3401d04: Deleting consensus metadata
I20260927 04:02:59.042761 19248 catalog_manager.cc:4841] TS d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703): tablet bba8d0b752714d7b87eac25ac379dc2e (table test-table [id=dd2492f0280b42a680aa9912a568ae8b]) successfully deleted
I20260927 04:02:59.042956 19249 catalog_manager.cc:4841] TS 7d7445c6f62343bf8279c4c4b3401d04 (127.17.184.193:39881): tablet bba8d0b752714d7b87eac25ac379dc2e (table test-table [id=dd2492f0280b42a680aa9912a568ae8b]) successfully deleted
I20260927 04:02:59.043463 19769 tablet_bootstrap.cc:492] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: No bootstrap required, opened a new log
I20260927 04:02:59.043529 19769 ts_tablet_manager.cc:1397] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260927 04:02:59.043668 19769 raft_consensus.cc:348] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.043748 19769 raft_consensus.cc:374] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:59.043771 19769 raft_consensus.cc:729] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0e2c48dff24502a950f3d18d6dc630, State: Initialized, Role: FOLLOWER
I20260927 04:02:59.043813 19769 consensus_queue.cc:260] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.043910 19769 ts_tablet_manager.cc:1428] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:59.047636 19780 raft_consensus.cc:482] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:59.047732 19780 raft_consensus.cc:504] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.047936 19780 leader_election.cc:288] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991), d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703)
I20260927 04:02:59.048110 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33afb6a855c4679a93d2f080694498e" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33d45be0bbc45c1abef3db02708cdad" is_pre_election: true
I20260927 04:02:59.048221 19699 raft_consensus.cc:2454] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 0.
I20260927 04:02:59.048151 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33afb6a855c4679a93d2f080694498e" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0e2c48dff24502a950f3d18d6dc630" is_pre_election: true
I20260927 04:02:59.048303 19559 raft_consensus.cc:2454] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 0.
I20260927 04:02:59.048457 19351 leader_election.cc:302] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0e2c48dff24502a950f3d18d6dc630, 7d7445c6f62343bf8279c4c4b3401d04; no voters:
I20260927 04:02:59.048549 19780 raft_consensus.cc:2781] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:59.048599 19780 raft_consensus.cc:482] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:59.048630 19780 raft_consensus.cc:3037] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:59.049520 19780 raft_consensus.cc:504] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.049659 19780 leader_election.cc:288] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 election: Requested vote from peers 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991), d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703)
I20260927 04:02:59.049803 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33afb6a855c4679a93d2f080694498e" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33d45be0bbc45c1abef3db02708cdad"
I20260927 04:02:59.049844 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33afb6a855c4679a93d2f080694498e" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0e2c48dff24502a950f3d18d6dc630"
I20260927 04:02:59.049883 19699 raft_consensus.cc:3037] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:59.049906 19559 raft_consensus.cc:3037] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:59.050725 19699 raft_consensus.cc:2454] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 1.
I20260927 04:02:59.050725 19559 raft_consensus.cc:2454] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 1.
I20260927 04:02:59.050928 19351 leader_election.cc:302] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0e2c48dff24502a950f3d18d6dc630, 7d7445c6f62343bf8279c4c4b3401d04; no voters:
I20260927 04:02:59.051025 19780 raft_consensus.cc:2781] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:59.051095 19780 raft_consensus.cc:686] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 LEADER]: Becoming Leader. State: Replica: 7d7445c6f62343bf8279c4c4b3401d04, State: Running, Role: LEADER
I20260927 04:02:59.051138 19780 consensus_queue.cc:237] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.051669 19262 catalog_manager.cc:5495] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d7445c6f62343bf8279c4c4b3401d04 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } health_report { overall_health: UNKNOWN } } }
W20260927 04:02:59.069099 19466 tablet.cc:2369] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 04:02:59 dist-test-slave-zqrn krb5kdc[19217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:02:59.079640 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.071773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.17.184.254:41697)
0927 04:02:59.071858 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:59.071860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:59.071993 (+ 133us) server_negotiation.cc:407] Connection header received
0927 04:02:59.072067 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:59.072067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:59.072077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:59.072129 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:59.072774 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.073270 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.074456 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.074640 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.074704 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.077038 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:59.077039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:59.077039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:59.077046 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:59.078427 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:59.078823 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:59.078824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:59.078825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:59.078888 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:59.079104 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:59.079105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:59.079105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:59.079218 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:59.079289 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:59.079443 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 04:02:59.079527 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 04:02:59.079746 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.071713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41697 (local address 127.0.0.1:34416)
0927 04:02:59.071908 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:59.071920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:59.072000 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:59.072154 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:59.072155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:59.072612 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:02:59.072671 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:59.072674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.073358 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.073359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:59.074329 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:59.074332 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.074722 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.074723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:59.074785 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.075411 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:02:59.075416 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:02:59.076936 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:02:59.078521 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:59.078529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:59.078531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:59.078735 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:59.078986 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:02:59.078986 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:02:59.078987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:02:59.079045 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:02:59.079289 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:02:59.079297 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:02:59.079340 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 04:02:59.079421 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 04:02:59.079601 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":81,"thread_start_us":116,"threads_started":1}
I20260927 04:02:59.081816 19262 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:34416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 04:02:59.082017 19262 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:02:59.083505 19399 tablet_service.cc:1467] Processing CreateTablet for tablet b5e5390f3bab41fc83e80bee60a4ce65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=520b54eb629e431ca45f6e92f7864c41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:02:59.083524 19539 tablet_service.cc:1467] Processing CreateTablet for tablet b5e5390f3bab41fc83e80bee60a4ce65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=520b54eb629e431ca45f6e92f7864c41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:02:59.083631 19399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e5390f3bab41fc83e80bee60a4ce65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:59.083670 19539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e5390f3bab41fc83e80bee60a4ce65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:59.084846 19770 tablet_bootstrap.cc:492] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: Bootstrap starting.
I20260927 04:02:59.085034 19662 tablet_service.cc:1467] Processing CreateTablet for tablet b5e5390f3bab41fc83e80bee60a4ce65 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=520b54eb629e431ca45f6e92f7864c41]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:02:59.085101 19769 tablet_bootstrap.cc:492] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: Bootstrap starting.
I20260927 04:02:59.085165 19662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5e5390f3bab41fc83e80bee60a4ce65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:02:59.085320 19770 tablet_bootstrap.cc:654] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:59.085522 19769 tablet_bootstrap.cc:654] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:59.086169 19769 tablet_bootstrap.cc:492] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: No bootstrap required, opened a new log
I20260927 04:02:59.086221 19769 ts_tablet_manager.cc:1397] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:02:59.086409 19770 tablet_bootstrap.cc:492] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: No bootstrap required, opened a new log
I20260927 04:02:59.086374 19769 raft_consensus.cc:348] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.086447 19769 raft_consensus.cc:374] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:59.086452 19770 ts_tablet_manager.cc:1397] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:02:59.086470 19769 raft_consensus.cc:729] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0e2c48dff24502a950f3d18d6dc630, State: Initialized, Role: FOLLOWER
I20260927 04:02:59.086512 19769 consensus_queue.cc:260] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.086596 19769 ts_tablet_manager.cc:1428] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:59.086584 19770 raft_consensus.cc:348] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.086647 19770 raft_consensus.cc:374] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:59.086668 19770 raft_consensus.cc:729] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7445c6f62343bf8279c4c4b3401d04, State: Initialized, Role: FOLLOWER
I20260927 04:02:59.086694 19768 tablet_bootstrap.cc:492] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: Bootstrap starting.
I20260927 04:02:59.086706 19770 consensus_queue.cc:260] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.086795 19770 ts_tablet_manager.cc:1428] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:02:59.086920 19780 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.086925 19776 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.086992 19780 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:02:59.087008 19776 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:02:59.087141 19768 tablet_bootstrap.cc:654] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: Neither blocks nor log segments found. Creating new log.
I20260927 04:02:59.087793 19768 tablet_bootstrap.cc:492] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: No bootstrap required, opened a new log
I20260927 04:02:59.087850 19768 ts_tablet_manager.cc:1397] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 04:02:59.088094 19768 raft_consensus.cc:348] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.088169 19768 raft_consensus.cc:374] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:02:59.088195 19768 raft_consensus.cc:729] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33d45be0bbc45c1abef3db02708cdad, State: Initialized, Role: FOLLOWER
I20260927 04:02:59.088235 19768 consensus_queue.cc:260] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.088328 19768 ts_tablet_manager.cc:1428] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:02:59.088358 19774 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.088425 19774 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 04:02:59.129735 19746 tablet.cc:2369] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:02:59.157429 19780 raft_consensus.cc:482] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:02:59.157527 19780 raft_consensus.cc:504] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.157842 19780 leader_election.cc:288] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991), d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703)
I20260927 04:02:59.158074 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e5390f3bab41fc83e80bee60a4ce65" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33d45be0bbc45c1abef3db02708cdad" is_pre_election: true
I20260927 04:02:59.158075 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e5390f3bab41fc83e80bee60a4ce65" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0e2c48dff24502a950f3d18d6dc630" is_pre_election: true
I20260927 04:02:59.158175 19699 raft_consensus.cc:2454] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 0.
I20260927 04:02:59.158192 19559 raft_consensus.cc:2454] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 0.
I20260927 04:02:59.158339 19351 leader_election.cc:302] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0e2c48dff24502a950f3d18d6dc630, 7d7445c6f62343bf8279c4c4b3401d04; no voters:
I20260927 04:02:59.158491 19780 raft_consensus.cc:2781] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:02:59.158572 19780 raft_consensus.cc:482] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:02:59.158598 19780 raft_consensus.cc:3037] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:59.159308 19780 raft_consensus.cc:504] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.159453 19780 leader_election.cc:288] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 election: Requested vote from peers 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991), d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703)
I20260927 04:02:59.159629 19559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e5390f3bab41fc83e80bee60a4ce65" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e0e2c48dff24502a950f3d18d6dc630"
I20260927 04:02:59.159735 19559 raft_consensus.cc:3037] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:59.159744 19699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5e5390f3bab41fc83e80bee60a4ce65" candidate_uuid: "7d7445c6f62343bf8279c4c4b3401d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d33d45be0bbc45c1abef3db02708cdad"
I20260927 04:02:59.159819 19699 raft_consensus.cc:3037] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:02:59.160545 19559 raft_consensus.cc:2454] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 1.
I20260927 04:02:59.160547 19699 raft_consensus.cc:2454] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7445c6f62343bf8279c4c4b3401d04 in term 1.
I20260927 04:02:59.160730 19352 leader_election.cc:302] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7445c6f62343bf8279c4c4b3401d04, d33d45be0bbc45c1abef3db02708cdad; no voters:
I20260927 04:02:59.160815 19780 raft_consensus.cc:2781] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:02:59.160919 19780 raft_consensus.cc:686] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 LEADER]: Becoming Leader. State: Replica: 7d7445c6f62343bf8279c4c4b3401d04, State: Running, Role: LEADER
I20260927 04:02:59.160975 19780 consensus_queue.cc:237] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } }
I20260927 04:02:59.161259 19788 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: TxnStatusTablet state changed. Reason: New leader 7d7445c6f62343bf8279c4c4b3401d04. Latest consensus state: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.161358 19788 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:02:59.161509 19810 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 04:02:59.161620 19810 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 04:02:59.161648 19262 catalog_manager.cc:5495] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d7445c6f62343bf8279c4c4b3401d04 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } health_report { overall_health: UNKNOWN } } }
I20260927 04:02:59.167439 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.164698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.17.184.193:39881)
0927 04:02:59.164755 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:59.164757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:59.164766 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:02:59.164802 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:59.164803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:59.164812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:59.164904 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:59.165045 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.165529 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.166607 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.166824 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.166884 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.167172 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:59.167217 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:59.167277 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:59.167366 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 04:02:59.167386 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:02:59.167495 19805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.164576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:39881 (local address 127.0.0.1:50034)
0927 04:02:59.164657 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:59.164672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:59.164727 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:59.164917 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:59.164918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:59.164920 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:02:59.164975 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:59.164977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.165608 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.165609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:59.166516 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:59.166519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.166942 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.166943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:59.167016 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.167035 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:02:59.167295 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:02:59.167305 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 04:02:59.167364 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 04:02:59.167414 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:02:59.168460 19699 raft_consensus.cc:1261] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Refusing update from remote peer 7d7445c6f62343bf8279c4c4b3401d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:59.168502 19559 raft_consensus.cc:1261] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Refusing update from remote peer 7d7445c6f62343bf8279c4c4b3401d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:59.168764 19788 consensus_queue.cc:1035] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:59.168938 19780 consensus_queue.cc:1035] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:59.169513 19774 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: TxnStatusTablet state changed. Reason: New leader 7d7445c6f62343bf8279c4c4b3401d04. Latest consensus state: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.169617 19774 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:02:59.169723 19776 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: TxnStatusTablet state changed. Reason: New leader 7d7445c6f62343bf8279c4c4b3401d04. Latest consensus state: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.169808 19776 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:02:59.170337 19788 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.170418 19788 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:02:59.171036 19776 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.171126 19776 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P 0e0e2c48dff24502a950f3d18d6dc630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:02:59.171068 19784 mvcc.cc:205] Tried to move back new op lower bound from 7333813367472480256 to 7333813367443783680. Current Snapshot: MvccSnapshot[applied={T|T < 7333813367472480256}]
I20260927 04:02:59.171411 19788 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.171478 19788 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P 7d7445c6f62343bf8279c4c4b3401d04: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:02:59.171868 19774 tablet_replica.cc:439] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d7445c6f62343bf8279c4c4b3401d04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7445c6f62343bf8279c4c4b3401d04" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 39881 } } peers { permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 } } peers { permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 } } }
I20260927 04:02:59.171950 19774 tablet_replica.cc:442] T b5e5390f3bab41fc83e80bee60a4ce65 P d33d45be0bbc45c1abef3db02708cdad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:02:59.177055 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.174237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:39881 (local address 127.17.184.193:58857)
0927 04:02:59.174455 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 04:02:59.174468 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:02:59.174525 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:02:59.174654 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:02:59.174655 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:02:59.174656 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:02:59.174704 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:59.174705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.175578 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.175579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:59.176207 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:02:59.176209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.176664 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.176665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:02:59.176716 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.176729 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:02:59.176900 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:02:59.176902 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:02:59.176936 (+ 34us) client_negotiation.cc:240] Negotiation successful
0927 04:02:59.176986 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260927 04:02:59.177054 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.174437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:58857 (local address 127.17.184.193:39881)
0927 04:02:59.174497 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:59.174499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:59.174511 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:02:59.174580 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:59.174581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:59.174590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:59.174652 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:02:59.174742 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.175491 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.176289 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.176583 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.176652 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.176807 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:02:59.176841 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:02:59.176893 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:59.176940 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 04:02:59.176960 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 04:02:59.179199 19559 raft_consensus.cc:1261] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Refusing update from remote peer 7d7445c6f62343bf8279c4c4b3401d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:59.179211 19699 raft_consensus.cc:1261] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Refusing update from remote peer 7d7445c6f62343bf8279c4c4b3401d04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:02:59.179400 19788 consensus_queue.cc:1035] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e0e2c48dff24502a950f3d18d6dc630" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:59.179488 19780 consensus_queue.cc:1035] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d33d45be0bbc45c1abef3db02708cdad" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:02:59.194191 19262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:59.194299 19262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:02:59.195099 19262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd00f909fc0e4950ae0439495d68053a: Sending DeleteTablet for 3 replicas of tablet b33afb6a855c4679a93d2f080694498e
I20260927 04:02:59.195475 19539 tablet_service.cc:1514] Processing DeleteTablet for tablet b33afb6a855c4679a93d2f080694498e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:59 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:39130
I20260927 04:02:59.195503 19662 tablet_service.cc:1514] Processing DeleteTablet for tablet b33afb6a855c4679a93d2f080694498e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:59 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:40564
I20260927 04:02:59.195585 19793 tablet_replica.cc:330] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: stopping tablet replica
I20260927 04:02:59.195612 19794 tablet_replica.cc:330] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: stopping tablet replica
I20260927 04:02:59.195643 19793 raft_consensus.cc:2229] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:59.195660 19794 raft_consensus.cc:2229] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:02:59.195684 19793 raft_consensus.cc:2258] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:59.195694 19794 raft_consensus.cc:2258] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:59.195866 19399 tablet_service.cc:1514] Processing DeleteTablet for tablet b33afb6a855c4679a93d2f080694498e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:02:59 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:50026
I20260927 04:02:59.195976 19795 tablet_replica.cc:330] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: stopping tablet replica
I20260927 04:02:59.196079 19795 raft_consensus.cc:2229] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:02:59.196146 19795 raft_consensus.cc:2258] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:02:59.196790 19795 ts_tablet_manager.cc:1905] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:59.196925 19793 ts_tablet_manager.cc:1905] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:59.198670 19793 ts_tablet_manager.cc:1918] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:02:59.198670 19795 ts_tablet_manager.cc:1918] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:02:59.198731 19795 log.cc:1196] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/b33afb6a855c4679a93d2f080694498e
I20260927 04:02:59.198731 19793 log.cc:1196] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/b33afb6a855c4679a93d2f080694498e
I20260927 04:02:59.199018 19793 ts_tablet_manager.cc:1939] T b33afb6a855c4679a93d2f080694498e P 0e0e2c48dff24502a950f3d18d6dc630: Deleting consensus metadata
I20260927 04:02:59.199054 19795 ts_tablet_manager.cc:1939] T b33afb6a855c4679a93d2f080694498e P 7d7445c6f62343bf8279c4c4b3401d04: Deleting consensus metadata
I20260927 04:02:59.199405 19247 catalog_manager.cc:4841] TS 0e0e2c48dff24502a950f3d18d6dc630 (127.17.184.194:37991): tablet b33afb6a855c4679a93d2f080694498e (table test-table [id=d7d9d60f0cbc4b3cb82af6d916703c31]) successfully deleted
I20260927 04:02:59.199405 19249 catalog_manager.cc:4841] TS 7d7445c6f62343bf8279c4c4b3401d04 (127.17.184.193:39881): tablet b33afb6a855c4679a93d2f080694498e (table test-table [id=d7d9d60f0cbc4b3cb82af6d916703c31]) successfully deleted
I20260927 04:02:59.199728 19794 ts_tablet_manager.cc:1905] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:02:59.201278 19794 ts_tablet_manager.cc:1918] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:02:59.201347 19794 log.cc:1196] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/b33afb6a855c4679a93d2f080694498e
I20260927 04:02:59.201576 19794 ts_tablet_manager.cc:1939] T b33afb6a855c4679a93d2f080694498e P d33d45be0bbc45c1abef3db02708cdad: Deleting consensus metadata
I20260927 04:02:59.202070 19248 catalog_manager.cc:4841] TS d33d45be0bbc45c1abef3db02708cdad (127.17.184.195:44703): tablet b33afb6a855c4679a93d2f080694498e (table test-table [id=d7d9d60f0cbc4b3cb82af6d916703c31]) successfully deleted
Sep 27 04:02:59 dist-test-slave-zqrn krb5kdc[19217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481778, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:02:59.207540 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.196757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50040 (local address 127.17.184.193:39881)
0927 04:02:59.196800 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:59.196803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:59.197060 (+ 257us) server_negotiation.cc:407] Connection header received
0927 04:02:59.197164 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:59.197165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:59.197175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:59.197208 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:59.198055 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.198888 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.200229 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.200597 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.200691 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.204394 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:59.204409 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:59.204409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:59.204423 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:59.206234 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:59.206629 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:59.206634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:59.206634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:59.206697 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:59.206927 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:59.206928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:59.206928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:59.207091 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:59.207278 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:59.207315 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 04:02:59.207398 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 04:02:59.207841 19459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50040
I20260927 04:02:59.216295 19312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:02:59.208902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34428 (local address 127.17.184.254:41697)
0927 04:02:59.208972 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:02:59.208974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:02:59.209036 (+ 62us) server_negotiation.cc:407] Connection header received
0927 04:02:59.209074 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:02:59.209075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:02:59.209089 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:02:59.209149 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:02:59.209619 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.210664 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.211531 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:02:59.211868 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:02:59.211962 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:02:59.213248 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:02:59.213249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:02:59.213250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:02:59.213259 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:02:59.215015 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:59.215374 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:59.215375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:59.215375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:59.215450 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:02:59.215712 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:02:59.215713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:02:59.215713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:02:59.215872 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:02:59.216000 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 04:02:59.216088 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 04:02:59.216179 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260927 04:02:59.216609 19262 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34428
I20260927 04:02:59.217345 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 19334
I20260927 04:02:59.224656 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 19474
I20260927 04:02:59.229485 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 19614
I20260927 04:02:59.234277 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 19230
2026-09-27T04:02:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2180 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:02:59 dist-test-slave-zqrn krb5kdc[19826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:02:59 dist-test-slave-zqrn krb5kdc[19826](info): set up 2 sockets
Sep 27 04:02:59 dist-test-slave-zqrn krb5kdc[19826](info): commencing operation
krb5kdc: starting...
W20260927 04:03:00.290725 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:00 dist-test-slave-zqrn krb5kdc[19826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:00.423055 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:39611
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:39611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:00.504374 19839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:00.504630 19839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:00.504665 19839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:00.505802 19839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:00.505873 19839 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 04:03:00.505894 19839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:00.505916 19839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:00.505945 19839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:00.505966 19839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:00.507445 19839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:39611
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:39611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:00.507733 19839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:00.507963 19839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:03:00.512948 19839 server_base.cc:1034] running on GCE node
W20260927 04:03:00.512964 19845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:00.513067 19847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:00.512948 19844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:00.513471 19839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:00.513847 19839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:00.514972 19839 hybrid_clock.cc:648] HybridClock initialized: now 1790481780514953 us; error 34 us; skew 500 ppm
Sep 27 04:03:00 dist-test-slave-zqrn krb5kdc[19826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481780, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:00.517760 19839 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:00.518236 19839 webserver.cc:460] Webserver started at http://127.17.184.254:43423/ using document root <none> and password file <none>
I20260927 04:03:00.518507 19839 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:00.518576 19839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:00.518733 19839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:00.519841 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "eb08e31c2d3847dd97a01bb54405e862"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "151b35603a29cd49739faf228a38afce"
server_key_iv: "c30b77775191d098d4616dde1aaa3025"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.520216 19839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "eb08e31c2d3847dd97a01bb54405e862"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "151b35603a29cd49739faf228a38afce"
server_key_iv: "c30b77775191d098d4616dde1aaa3025"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.521552 19839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:00.522392 19854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:00.522621 19839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:00.522774 19839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "eb08e31c2d3847dd97a01bb54405e862"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "151b35603a29cd49739faf228a38afce"
server_key_iv: "c30b77775191d098d4616dde1aaa3025"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.522858 19839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:00.533636 19839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:00.535952 19839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:00.536131 19839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:00.540998 19839 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:39611
I20260927 04:03:00.541054 19916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:39611 every 8 connection(s)
I20260927 04:03:00.541380 19839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:00.542040 19917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:00.544420 19917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: Bootstrap starting.
I20260927 04:03:00.545018 19917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:00.545305 19917 log.cc:824] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:00.546008 19917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: No bootstrap required, opened a new log
I20260927 04:03:00.548120 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 19839
I20260927 04:03:00.548210 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:00.548378 18147 external_mini_cluster.cc:1428] Setting key 3f311f4a1003e76359b58508a01285e4
I20260927 04:03:00.548401 19917 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08e31c2d3847dd97a01bb54405e862" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39611 } }
I20260927 04:03:00.548586 19917 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:00.548615 19917 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb08e31c2d3847dd97a01bb54405e862, State: Initialized, Role: FOLLOWER
I20260927 04:03:00.548796 19917 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08e31c2d3847dd97a01bb54405e862" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39611 } }
I20260927 04:03:00.548883 19917 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:00.548957 19917 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:00.549072 19917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:00.549638 19917 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08e31c2d3847dd97a01bb54405e862" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39611 } }
I20260927 04:03:00.549767 19917 leader_election.cc:302] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb08e31c2d3847dd97a01bb54405e862; no voters:
I20260927 04:03:00.549957 19917 leader_election.cc:288] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:00.549978 19922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:00.550087 19922 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [term 1 LEADER]: Becoming Leader. State: Replica: eb08e31c2d3847dd97a01bb54405e862, State: Running, Role: LEADER
I20260927 04:03:00.550239 19917 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:00.550221 19922 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08e31c2d3847dd97a01bb54405e862" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39611 } }
I20260927 04:03:00.550729 19924 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb08e31c2d3847dd97a01bb54405e862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08e31c2d3847dd97a01bb54405e862" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39611 } } }
I20260927 04:03:00.550745 19923 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb08e31c2d3847dd97a01bb54405e862. Latest consensus state: current_term: 1 leader_uuid: "eb08e31c2d3847dd97a01bb54405e862" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb08e31c2d3847dd97a01bb54405e862" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39611 } } }
I20260927 04:03:00.550829 19924 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:00.550851 19923 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:00.551708 19929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:00.552294 19929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:00.554014 19929 catalog_manager.cc:1347] Generated new cluster ID: 787b288780a345819d8074c8b9ad2ca7
I20260927 04:03:00.554068 19929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:03:00 dist-test-slave-zqrn krb5kdc[19826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:00.560279 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.548883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.17.184.254:39611)
0927 04:03:00.549062 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:00.549072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:00.549109 (+ 37us) server_negotiation.cc:407] Connection header received
0927 04:03:00.549252 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:00.549258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:00.549310 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:00.549449 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:00.550217 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.551650 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.552794 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.553347 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.553498 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.556461 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:00.556465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:00.556466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:00.556478 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:00.558513 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.558943 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.558946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.558946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.559077 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.559315 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.559316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.559316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.559643 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:00.559833 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:00.559907 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 04:03:00.560056 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":89,"threads_started":1}
I20260927 04:03:00.563961 19929 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:00.564484 19929 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:00.569063 19929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: Generated new TSK 0
I20260927 04:03:00.569257 19929 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:03:00.602229 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:39611
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:00.685052 19943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:00.685297 19943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:00.685338 19943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:00.686512 19943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:00.686579 19943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:00.686617 19943 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 04:03:00.686664 19943 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:03:00.686703 19943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:00.688252 19943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:39611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:00.688529 19943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:00.688772 19943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:00.694015 19949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:00.694043 19951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:00.694073 19948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:00.694139 19943 server_base.cc:1034] running on GCE node
I20260927 04:03:00.694473 19943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:00.694675 19943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:00.695837 19943 hybrid_clock.cc:648] HybridClock initialized: now 1790481780695818 us; error 33 us; skew 500 ppm
Sep 27 04:03:00 dist-test-slave-zqrn krb5kdc[19826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481780, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:00.698398 19943 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:03:00.698853 19943 webserver.cc:460] Webserver started at http://127.17.184.193:46717/ using document root <none> and password file <none>
I20260927 04:03:00.699122 19943 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:00.699203 19943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:00.699357 19943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:00.700461 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "e3c37a3c630045109e15abf6ee8a59a3"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "faab1441e50cd7603d51157eed47f26e"
server_key_iv: "b0b57e5ce89fd31c06ed30786d8ab95c"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.700868 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "e3c37a3c630045109e15abf6ee8a59a3"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "faab1441e50cd7603d51157eed47f26e"
server_key_iv: "b0b57e5ce89fd31c06ed30786d8ab95c"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.702255 19943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:00.703063 19958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:00.703248 19943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:00.703347 19943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "e3c37a3c630045109e15abf6ee8a59a3"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "faab1441e50cd7603d51157eed47f26e"
server_key_iv: "b0b57e5ce89fd31c06ed30786d8ab95c"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.703435 19943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:00.720341 19943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:00.723033 19943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:00.723244 19943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:00.723845 19943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:00.723918 19943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:00.723968 19943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:00.723991 19943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:03:00 dist-test-slave-zqrn krb5kdc[19826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481780, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:00.731626 19943 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:42299
I20260927 04:03:00.731689 20073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:42299 every 8 connection(s)
I20260927 04:03:00.732100 19943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:00.735249 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.724462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:39611 (local address 127.17.184.193:47767)
0927 04:03:00.724642 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:00.724680 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:00.725198 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:00.725513 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:00.725517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:00.725916 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:00.726286 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:00.726292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.727429 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.727430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:00.728394 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:00.728396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.728980 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.728981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:00.729126 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.729683 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:00.729691 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:00.731220 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:00.733155 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:00.733162 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:00.733163 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:00.733511 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:00.733900 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:00.733901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:00.733901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:00.734032 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:00.734409 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:00.734415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:00.734548 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 04:03:00.734650 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 04:03:00.735055 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260927 04:03:00.735272 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.724463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:47767 (local address 127.17.184.254:39611)
0927 04:03:00.724521 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:00.724524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:00.724760 (+ 236us) server_negotiation.cc:407] Connection header received
0927 04:03:00.725287 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:00.725289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:00.725308 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:00.725384 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:00.726400 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.727327 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.728469 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.728853 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.728945 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.731339 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:00.731341 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:00.731341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:00.731350 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:00.733073 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.733647 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.733648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.733648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.733729 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.734159 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.734160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.734160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.734298 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:00.734370 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:00.734738 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 04:03:00.735191 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 04:03:00.735922 20074 heartbeater.cc:347] Connected to a master server at 127.17.184.254:39611
I20260927 04:03:00.736014 20074 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:00.736235 20074 heartbeater.cc:511] Master 127.17.184.254:39611 requested a full tablet report, sending...
I20260927 04:03:00.736893 19871 ts_manager.cc:194] Registered new tserver with Master: e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299)
I20260927 04:03:00.737772 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 19943
I20260927 04:03:00.737891 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:00.737874 19871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:47767
I20260927 04:03:00.738086 18147 external_mini_cluster.cc:1428] Setting key d0813e6bcf26fd4a177b3f54c76dd844
I20260927 04:03:00.744014 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.737197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:39611 (local address 127.17.184.193:41949)
0927 04:03:00.737358 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:00.737369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:00.737438 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:00.737560 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:00.737560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:00.737763 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:00.737835 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:00.737837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.738706 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.738707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:00.739301 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:00.739303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.739984 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.739985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:00.740067 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.740614 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:00.740619 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:00.740930 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:00.742825 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:00.742826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:00.742827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:00.743080 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:00.743322 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:00.743323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:00.743323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:00.743390 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:00.743659 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:00.743660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:00.743713 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 04:03:00.743792 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 04:03:00.743909 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260927 04:03:00.744009 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.737297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:41949 (local address 127.17.184.254:39611)
0927 04:03:00.737344 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:00.737346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:00.737425 (+ 79us) server_negotiation.cc:407] Connection header received
0927 04:03:00.737494 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:00.737495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:00.737503 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:00.737547 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:00.737931 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.738596 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.739556 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.739868 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.739952 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.741029 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:00.741031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:00.741031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:00.741040 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:00.742691 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.743174 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.743175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.743184 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.743250 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.743463 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.743464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.743464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.743588 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:00.743669 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:00.743820 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 04:03:00.743907 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:03:00.767248 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:39611
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:00.854555 20083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:00.854791 20083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:00.854820 20083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:00.856135 20083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:00.856190 20083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:00.856215 20083 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 04:03:00.856294 20083 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:03:00.856324 20083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:00.857854 20083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:39611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:00.858100 20083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:00.858309 20083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:00.863955 20091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:00.863977 20089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:00.864038 20088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:00.864317 20083 server_base.cc:1034] running on GCE node
I20260927 04:03:00.864529 20083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:00.864799 20083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:00.866001 20083 hybrid_clock.cc:648] HybridClock initialized: now 1790481780865982 us; error 29 us; skew 500 ppm
Sep 27 04:03:00 dist-test-slave-zqrn krb5kdc[19826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481780, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:00.868857 20083 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:03:00.869414 20083 webserver.cc:460] Webserver started at http://127.17.184.194:35867/ using document root <none> and password file <none>
I20260927 04:03:00.869689 20083 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:00.869769 20083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:00.869942 20083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:00.871234 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "e18ae88d43f44d76a99c837f450fe0bb"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "d9dc5b030bf83f30f8c688a81b2e7dff"
server_key_iv: "9452d84ecaf524bb24d107dd522ece38"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.871613 20083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "e18ae88d43f44d76a99c837f450fe0bb"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "d9dc5b030bf83f30f8c688a81b2e7dff"
server_key_iv: "9452d84ecaf524bb24d107dd522ece38"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.873196 20083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:00.874089 20098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:00.874294 20083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:00.874428 20083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "e18ae88d43f44d76a99c837f450fe0bb"
format_stamp: "Formatted at 2026-09-27 04:03:00 on dist-test-slave-zqrn"
server_key: "d9dc5b030bf83f30f8c688a81b2e7dff"
server_key_iv: "9452d84ecaf524bb24d107dd522ece38"
server_key_version: "encryptionkey@0"
I20260927 04:03:00.874531 20083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:00.884498 20083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:00.887238 20083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:00.887413 20083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:00.888103 20083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:00.888165 20083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:00.888207 20083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:00.888233 20083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:03:00 dist-test-slave-zqrn krb5kdc[19826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481780, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:00.896548 20083 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:35469
I20260927 04:03:00.896646 20213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:35469 every 8 connection(s)
I20260927 04:03:00.896988 20083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:00.899394 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.888635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:48647 (local address 127.17.184.254:39611)
0927 04:03:00.888708 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:00.888711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:00.888972 (+ 261us) server_negotiation.cc:407] Connection header received
0927 04:03:00.889444 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:00.889446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:00.889461 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:00.889569 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:00.890259 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.891086 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.892184 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.892440 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.892505 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.896227 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:00.896228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:00.896229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:00.896238 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:00.897903 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.898371 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.898373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.898373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.898441 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.898688 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.898690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.898690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.898823 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:00.898902 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:00.899218 (+ 316us) server_negotiation.cc:299] Negotiation successful
0927 04:03:00.899297 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:03:00.899595 20107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.888614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:39611 (local address 127.17.184.194:48647)
0927 04:03:00.888831 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:00.888873 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:00.889344 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:00.889672 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:00.889675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:00.889914 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:00.890139 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:00.890145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.891187 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.891188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:00.892064 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:00.892067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.892543 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.892545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:00.892844 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.893511 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:00.893525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:00.896110 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:00.898006 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:00.898012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:00.898015 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:00.898287 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:00.898509 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:00.898510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:00.898511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:00.898617 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:00.898913 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:00.898919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:00.899066 (+ 147us) client_negotiation.cc:769] Sending connection context
0927 04:03:00.899185 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 04:03:00.899361 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260927 04:03:00.900225 20214 heartbeater.cc:347] Connected to a master server at 127.17.184.254:39611
I20260927 04:03:00.900350 20214 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:00.900537 20214 heartbeater.cc:511] Master 127.17.184.254:39611 requested a full tablet report, sending...
I20260927 04:03:00.900992 19870 ts_manager.cc:194] Registered new tserver with Master: e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469)
I20260927 04:03:00.901532 19870 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:48647
I20260927 04:03:00.902745 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 20083
I20260927 04:03:00.902855 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:00.903064 18147 external_mini_cluster.cc:1428] Setting key f3f6712921d2151ad2eca282310457d5
I20260927 04:03:00.908213 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.901406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:50143 (local address 127.17.184.254:39611)
0927 04:03:00.901463 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:00.901465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:00.901601 (+ 136us) server_negotiation.cc:407] Connection header received
0927 04:03:00.901652 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:00.901653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:00.901666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:00.901735 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:00.902121 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.902635 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.903847 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.904148 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.904227 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.905193 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:00.905195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:00.905195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:00.905203 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:00.906787 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.907264 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.907265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.907266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.907331 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:00.907590 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:00.907591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:00.907592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:00.907776 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:00.907844 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:00.908059 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 04:03:00.908136 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 04:03:00.908293 20220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:00.901340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:39611 (local address 127.17.184.194:50143)
0927 04:03:00.901526 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:00.901535 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:00.901586 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:00.901740 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:00.901740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:00.901972 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:00.902037 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:00.902040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.902782 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.902783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:00.903713 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:00.903715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:00.904245 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:00.904247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:00.904316 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:00.904815 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:00.904819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:00.905103 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:00.906878 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:00.906879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:00.906880 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:00.907147 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:00.907405 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:00.907406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:00.907406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:00.907474 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:00.907859 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:00.907861 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:00.907915 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 04:03:00.907998 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 04:03:00.908120 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:03:00.932179 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:39611
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:01.014986 20223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:01.015328 20223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:01.015380 20223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:01.016726 20223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:01.016788 20223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:01.016821 20223 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 04:03:01.016848 20223 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:03:01.016891 20223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:01.018393 20223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:39611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:01.018680 20223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:01.018944 20223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:01.024570 20228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:01.024588 20229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:01.024570 20231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:01.025291 20223 server_base.cc:1034] running on GCE node
I20260927 04:03:01.025516 20223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:01.025843 20223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:01.027011 20223 hybrid_clock.cc:648] HybridClock initialized: now 1790481781026988 us; error 39 us; skew 500 ppm
Sep 27 04:03:01 dist-test-slave-zqrn krb5kdc[19826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481781, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:01.029841 20223 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:03:01.030359 20223 webserver.cc:460] Webserver started at http://127.17.184.195:46643/ using document root <none> and password file <none>
I20260927 04:03:01.030614 20223 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:01.030675 20223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:01.030846 20223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:01.032043 20223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "206c99b0dda24e049ad95d4b8c6bca5e"
format_stamp: "Formatted at 2026-09-27 04:03:01 on dist-test-slave-zqrn"
server_key: "ff6601a46ecf9639c435ae56e2bc9bb0"
server_key_iv: "07e26c7ea7d91f14bbd2f349bc6ab0a7"
server_key_version: "encryptionkey@0"
I20260927 04:03:01.032464 20223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "206c99b0dda24e049ad95d4b8c6bca5e"
format_stamp: "Formatted at 2026-09-27 04:03:01 on dist-test-slave-zqrn"
server_key: "ff6601a46ecf9639c435ae56e2bc9bb0"
server_key_iv: "07e26c7ea7d91f14bbd2f349bc6ab0a7"
server_key_version: "encryptionkey@0"
I20260927 04:03:01.034058 20223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:01.035115 20238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:01.035313 20223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:01.035413 20223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "206c99b0dda24e049ad95d4b8c6bca5e"
format_stamp: "Formatted at 2026-09-27 04:03:01 on dist-test-slave-zqrn"
server_key: "ff6601a46ecf9639c435ae56e2bc9bb0"
server_key_iv: "07e26c7ea7d91f14bbd2f349bc6ab0a7"
server_key_version: "encryptionkey@0"
I20260927 04:03:01.035516 20223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:01.068233 20223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:01.070600 20223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:01.070851 20223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:01.071496 20223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:01.071563 20223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:01.071610 20223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:01.071674 20223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:03:01 dist-test-slave-zqrn krb5kdc[19826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481781, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:01.079439 20223 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:37119
I20260927 04:03:01.079595 20353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:37119 every 8 connection(s)
I20260927 04:03:01.079897 20223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:01.083009 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.071988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:53213 (local address 127.17.184.254:39611)
0927 04:03:01.072031 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.072034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.072264 (+ 230us) server_negotiation.cc:407] Connection header received
0927 04:03:01.072871 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.072872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.072886 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.072948 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:01.073839 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.074641 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.076567 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.076846 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.076944 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.079957 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:01.079959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:01.079960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:01.079968 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:01.081560 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:01.081992 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:01.081993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:01.081994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:01.082055 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:01.082319 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:01.082320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:01.082321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:01.082444 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:01.082519 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.082798 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.082911 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:03:01.083214 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.071994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:39611 (local address 127.17.184.195:53213)
0927 04:03:01.072142 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.072179 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.072726 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.073047 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.073052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.073446 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:01.073744 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.073751 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.074767 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.074768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.076453 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.076457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.076987 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.076988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.077171 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.077734 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:01.077745 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:01.079856 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:01.081666 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:01.081672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:01.081675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:01.081910 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:01.082129 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:01.082130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:01.082130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:01.082247 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:01.082511 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:01.082517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:01.082660 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.082775 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.082975 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":87,"threads_started":1}
I20260927 04:03:01.083698 20354 heartbeater.cc:347] Connected to a master server at 127.17.184.254:39611
I20260927 04:03:01.083782 20354 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:01.083976 20354 heartbeater.cc:511] Master 127.17.184.254:39611 requested a full tablet report, sending...
I20260927 04:03:01.084319 19871 ts_manager.cc:194] Registered new tserver with Master: 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:01.084936 19871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:53213
I20260927 04:03:01.087708 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 20223
I20260927 04:03:01.087809 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:01.087978 18147 external_mini_cluster.cc:1428] Setting key d54c2b8e44e5bc13ee1f847cc896b19a
I20260927 04:03:01.089145 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:03:01.091004 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.084909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:53567 (local address 127.17.184.254:39611)
0927 04:03:01.084951 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.084953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.085051 (+ 98us) server_negotiation.cc:407] Connection header received
0927 04:03:01.085162 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.085164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.085174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.085234 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:01.085542 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.086085 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.086840 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.087093 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.087146 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.088221 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:01.088222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:01.088223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:01.088230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:01.089879 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:01.090207 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:01.090208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:01.090209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:01.090275 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:01.090469 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:01.090470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:01.090471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:01.090588 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:01.090662 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.090816 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.090913 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:03:01.091039 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.084818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:39611 (local address 127.17.184.195:53567)
0927 04:03:01.084973 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.084986 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.085081 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.085241 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.085242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.085418 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:01.085470 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.085471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.086190 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.086191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.086740 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.086741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.087194 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.087195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.087257 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.087680 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:01.087685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:01.088064 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:01.089965 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:01.089966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:01.089966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:01.090129 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:01.090346 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:01.090347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:01.090347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:01.090396 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:01.090655 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:01.090656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:01.090701 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.090791 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.090942 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
Sep 27 04:03:01 dist-test-slave-zqrn krb5kdc[19826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:03:01 dist-test-slave-zqrn krb5kdc[19826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:01.115054 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.107603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.17.184.254:39611)
0927 04:03:01.107672 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.107674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.107685 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:01.107722 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.107723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.107733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.107790 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:01.108409 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.108978 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.109733 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.109935 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.110046 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.112414 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:01.112416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:01.112417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:01.112425 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:01.113826 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:01.114194 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:01.114195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:01.114196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:01.114257 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:01.114461 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:01.114462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:01.114462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:01.114583 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:01.114691 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.114842 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.114944 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:03:01.116612 19871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:01.118251 19871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:01.125494 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.121150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.17.184.194:35469)
0927 04:03:01.121327 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.121331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.121479 (+ 148us) server_negotiation.cc:407] Connection header received
0927 04:03:01.121528 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.121530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.121548 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.121655 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.121857 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.123179 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.124359 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.125061 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.125134 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.125184 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.125336 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.125379 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":71,"threads_started":1}
I20260927 04:03:01.125487 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.120847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:37119 (local address 127.0.0.1:41248)
0927 04:03:01.121143 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.121191 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.121265 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.121823 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.121824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.121826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.121876 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.121877 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.122743 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.122744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.123993 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.123995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.125118 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.125120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.125215 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.125225 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.125310 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.125382 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":25,"thread_start_us":117,"threads_started":1}
I20260927 04:03:01.125808 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.121106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41248 (local address 127.17.184.195:37119)
0927 04:03:01.121645 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.121649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.121658 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:01.121694 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.121696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.121709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.121820 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.121932 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.122631 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.124087 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.124880 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.125007 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.125080 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.125646 (+ 566us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.125692 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":87,"threads_started":1}
I20260927 04:03:01.125487 20373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.120859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35469 (local address 127.0.0.1:58094)
0927 04:03:01.121376 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.121385 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.121437 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.121729 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.121730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.121732 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.121772 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.121774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.123270 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.123271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.124268 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.124270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.125129 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.125130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.125209 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.125225 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.125310 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.125379 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":106,"threads_started":1}
I20260927 04:03:01.125883 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.120845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:42299 (local address 127.0.0.1:52884)
0927 04:03:01.121143 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.121191 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.121265 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.121533 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.121536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.121538 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.121622 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.121624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.123321 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.123322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.124558 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.124561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.125618 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.125619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.125691 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.125695 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.125746 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.125787 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":52,"threads_started":1}
I20260927 04:03:01.126441 20288 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98e3b89a404b47867197cdf021a1f3 (DEFAULT_TABLE table=test-table [id=85af126bd1f14f58b796da4b7982aad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:01.126627 20148 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98e3b89a404b47867197cdf021a1f3 (DEFAULT_TABLE table=test-table [id=85af126bd1f14f58b796da4b7982aad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:01.126796 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98e3b89a404b47867197cdf021a1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.127040 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.120970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.17.184.193:42299)
0927 04:03:01.121395 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.121398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.121406 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:01.121438 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.121439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.121452 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.121519 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.121964 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.123228 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.124651 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.125457 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.125529 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.125578 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.125972 (+ 394us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.126012 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":82,"threads_started":1}
I20260927 04:03:01.127044 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98e3b89a404b47867197cdf021a1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.129449 20377 tablet_bootstrap.cc:492] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Bootstrap starting.
I20260927 04:03:01.130201 20378 tablet_bootstrap.cc:492] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Bootstrap starting.
I20260927 04:03:01.130700 20377 tablet_bootstrap.cc:654] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.130923 20378 tablet_bootstrap.cc:654] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.131124 20377 log.cc:824] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:01.131011 20008 tablet_service.cc:1467] Processing CreateTablet for tablet 1c98e3b89a404b47867197cdf021a1f3 (DEFAULT_TABLE table=test-table [id=85af126bd1f14f58b796da4b7982aad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:01.131268 20378 log.cc:824] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:01.131381 20008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c98e3b89a404b47867197cdf021a1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.131973 20377 tablet_bootstrap.cc:492] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: No bootstrap required, opened a new log
I20260927 04:03:01.132014 20378 tablet_bootstrap.cc:492] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: No bootstrap required, opened a new log
I20260927 04:03:01.132050 20377 ts_tablet_manager.cc:1397] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 04:03:01.132079 20378 ts_tablet_manager.cc:1397] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:01.134141 20381 tablet_bootstrap.cc:492] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Bootstrap starting.
I20260927 04:03:01.134336 20377 raft_consensus.cc:348] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.134336 20378 raft_consensus.cc:348] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.134514 20378 raft_consensus.cc:374] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.134514 20377 raft_consensus.cc:374] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.134553 20378 raft_consensus.cc:729] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18ae88d43f44d76a99c837f450fe0bb, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.134569 20377 raft_consensus.cc:729] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206c99b0dda24e049ad95d4b8c6bca5e, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.134648 20381 tablet_bootstrap.cc:654] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.134694 20378 consensus_queue.cc:260] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.134694 20377 consensus_queue.cc:260] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.135010 20377 ts_tablet_manager.cc:1428] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:03:01.135010 20378 ts_tablet_manager.cc:1428] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:03:01.135066 20354 heartbeater.cc:503] Master 127.17.184.254:39611 was elected leader, sending a full tablet report...
I20260927 04:03:01.135154 20214 heartbeater.cc:503] Master 127.17.184.254:39611 was elected leader, sending a full tablet report...
I20260927 04:03:01.135390 20381 log.cc:824] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:01.136346 20381 tablet_bootstrap.cc:492] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: No bootstrap required, opened a new log
I20260927 04:03:01.136420 20381 ts_tablet_manager.cc:1397] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:01.137782 20381 raft_consensus.cc:348] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.137943 20381 raft_consensus.cc:374] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.138011 20381 raft_consensus.cc:729] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c37a3c630045109e15abf6ee8a59a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.138128 20381 consensus_queue.cc:260] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.138394 20381 ts_tablet_manager.cc:1428] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:01.138437 20074 heartbeater.cc:503] Master 127.17.184.254:39611 was elected leader, sending a full tablet report...
W20260927 04:03:01.147619 20215 tablet.cc:2369] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:01.231902 20382 raft_consensus.cc:482] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:01.232031 20382 raft_consensus.cc:504] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.232475 20382 leader_election.cc:288] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299), e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469)
W20260927 04:03:01.233536 20075 tablet.cc:2369] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:01.235590 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.232728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:34341 (local address 127.17.184.193:42299)
0927 04:03:01.232771 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.232773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.232784 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:01.232858 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.232858 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.232869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.232936 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.233158 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.233861 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.234824 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.235227 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.235285 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.235292 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.235454 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.235498 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:03:01.235599 20247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.232680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:42299 (local address 127.17.184.195:34341)
0927 04:03:01.232737 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.232749 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.232799 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.232972 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.232973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.232976 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.233052 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.233053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.233942 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.233943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.234716 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.234718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.235303 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.235304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.235391 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.235395 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.235449 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.235509 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:03:01.235694 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.232820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:33899 (local address 127.17.184.194:35469)
0927 04:03:01.232869 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.232871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.233007 (+ 136us) server_negotiation.cc:407] Connection header received
0927 04:03:01.233037 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.233038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.233048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.233106 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.233292 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.233879 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.234852 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.235304 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.235376 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.235384 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.235573 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.235621 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:03:01.235925 20386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.232716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35469 (local address 127.17.184.195:33899)
0927 04:03:01.232905 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.232915 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.232964 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.233142 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.233143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.233145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.233214 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.233215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.233994 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.233995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.234724 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.234726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.235399 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.235400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.235480 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.235484 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.235769 (+ 285us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.235821 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260927 04:03:01.235992 20025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98e3b89a404b47867197cdf021a1f3" candidate_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c37a3c630045109e15abf6ee8a59a3" is_pre_election: true
I20260927 04:03:01.236176 20025 raft_consensus.cc:2454] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 206c99b0dda24e049ad95d4b8c6bca5e in term 0.
I20260927 04:03:01.236234 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98e3b89a404b47867197cdf021a1f3" candidate_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e18ae88d43f44d76a99c837f450fe0bb" is_pre_election: true
I20260927 04:03:01.236382 20168 raft_consensus.cc:2454] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 206c99b0dda24e049ad95d4b8c6bca5e in term 0.
I20260927 04:03:01.236418 20242 leader_election.cc:302] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206c99b0dda24e049ad95d4b8c6bca5e, e3c37a3c630045109e15abf6ee8a59a3; no voters:
I20260927 04:03:01.236539 20382 raft_consensus.cc:2781] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:01.236606 20382 raft_consensus.cc:482] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:01.236629 20382 raft_consensus.cc:3037] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.237427 20382 raft_consensus.cc:504] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.237569 20382 leader_election.cc:288] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [CANDIDATE]: Term 1 election: Requested vote from peers e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299), e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469)
I20260927 04:03:01.237718 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98e3b89a404b47867197cdf021a1f3" candidate_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e18ae88d43f44d76a99c837f450fe0bb"
I20260927 04:03:01.237754 20025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c98e3b89a404b47867197cdf021a1f3" candidate_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c37a3c630045109e15abf6ee8a59a3"
I20260927 04:03:01.237818 20025 raft_consensus.cc:3037] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.237812 20168 raft_consensus.cc:3037] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.238723 20168 raft_consensus.cc:2454] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 206c99b0dda24e049ad95d4b8c6bca5e in term 1.
I20260927 04:03:01.238723 20025 raft_consensus.cc:2454] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 206c99b0dda24e049ad95d4b8c6bca5e in term 1.
I20260927 04:03:01.239253 20240 leader_election.cc:302] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206c99b0dda24e049ad95d4b8c6bca5e, e18ae88d43f44d76a99c837f450fe0bb; no voters:
I20260927 04:03:01.239363 20382 raft_consensus.cc:2781] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:01.239528 20382 raft_consensus.cc:686] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 LEADER]: Becoming Leader. State: Replica: 206c99b0dda24e049ad95d4b8c6bca5e, State: Running, Role: LEADER
I20260927 04:03:01.239637 20382 consensus_queue.cc:237] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } }
I20260927 04:03:01.240520 19869 catalog_manager.cc:5495] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195). New cstate: current_term: 1 leader_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } health_report { overall_health: UNKNOWN } } }
Sep 27 04:03:01 dist-test-slave-zqrn krb5kdc[19826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481780, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:01.267506 19921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.259730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.17.184.254:39611)
0927 04:03:01.259810 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.259813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.260022 (+ 209us) server_negotiation.cc:407] Connection header received
0927 04:03:01.260051 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.260051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.260062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.260140 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:01.260655 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.261164 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.261926 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.262140 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.262215 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.264695 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:01.264696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:01.264696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:01.264703 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:01.266070 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:01.266621 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:01.266622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:01.266622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:01.266675 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:01.266997 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:01.266998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:01.266999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:01.267115 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:01.267178 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.267359 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.267438 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:03:01.267623 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.259678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:39611 (local address 127.0.0.1:42794)
0927 04:03:01.259916 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.259927 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.259998 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.260156 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.260156 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.260476 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:01.260528 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.260531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.261260 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.261261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.261824 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.261826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.262259 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.262260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.262310 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.262984 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:01.262988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:01.264547 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:01.266221 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:01.266229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:01.266231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:01.266549 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:01.266771 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:01.266772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:01.266773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:01.266848 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:01.267184 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:01.267213 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:01.267262 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.267342 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.267521 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":67,"thread_start_us":161,"threads_started":1}
I20260927 04:03:01.270005 19869 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:42794:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 04:03:01.270224 19869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:01.271843 20148 tablet_service.cc:1467] Processing CreateTablet for tablet d6f3df6c23da4f3382119215411d6ddf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a204572b0f034fac9b7b573f2f1a194f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:03:01.271863 20288 tablet_service.cc:1467] Processing CreateTablet for tablet d6f3df6c23da4f3382119215411d6ddf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a204572b0f034fac9b7b573f2f1a194f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:03:01.271842 20008 tablet_service.cc:1467] Processing CreateTablet for tablet d6f3df6c23da4f3382119215411d6ddf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a204572b0f034fac9b7b573f2f1a194f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:03:01.272027 20008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f3df6c23da4f3382119215411d6ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.272027 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f3df6c23da4f3382119215411d6ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.272030 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f3df6c23da4f3382119215411d6ddf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.273536 20378 tablet_bootstrap.cc:492] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: Bootstrap starting.
I20260927 04:03:01.273577 20377 tablet_bootstrap.cc:492] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: Bootstrap starting.
I20260927 04:03:01.273583 20381 tablet_bootstrap.cc:492] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: Bootstrap starting.
I20260927 04:03:01.273933 20381 tablet_bootstrap.cc:654] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.274096 20377 tablet_bootstrap.cc:654] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.274221 20378 tablet_bootstrap.cc:654] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.274628 20381 tablet_bootstrap.cc:492] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: No bootstrap required, opened a new log
I20260927 04:03:01.274688 20381 ts_tablet_manager.cc:1397] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:01.274840 20381 raft_consensus.cc:348] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.274974 20381 raft_consensus.cc:374] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.275007 20381 raft_consensus.cc:729] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c37a3c630045109e15abf6ee8a59a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.275010 20377 tablet_bootstrap.cc:492] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: No bootstrap required, opened a new log
I20260927 04:03:01.275064 20377 ts_tablet_manager.cc:1397] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:01.275048 20381 consensus_queue.cc:260] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.275152 20381 ts_tablet_manager.cc:1428] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:01.275393 20385 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.275496 20385 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:01.275221 20377 raft_consensus.cc:348] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.275635 20377 raft_consensus.cc:374] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.275662 20377 raft_consensus.cc:729] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206c99b0dda24e049ad95d4b8c6bca5e, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.275709 20377 consensus_queue.cc:260] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.275835 20377 ts_tablet_manager.cc:1428] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:01.275951 20382 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.276031 20382 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:01.276367 20378 tablet_bootstrap.cc:492] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: No bootstrap required, opened a new log
I20260927 04:03:01.276453 20378 ts_tablet_manager.cc:1397] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 04:03:01.276592 20378 raft_consensus.cc:348] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.276659 20378 raft_consensus.cc:374] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.276674 20378 raft_consensus.cc:729] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18ae88d43f44d76a99c837f450fe0bb, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.276727 20378 consensus_queue.cc:260] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.276826 20378 ts_tablet_manager.cc:1428] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:01.276889 20383 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.276988 20383 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 04:03:01.330925 20355 tablet.cc:2369] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:01.557853 20383 raft_consensus.cc:482] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:01.557958 20383 raft_consensus.cc:504] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.558317 20383 leader_election.cc:288] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299), 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:01.561408 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.558612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:47825 (local address 127.17.184.193:42299)
0927 04:03:01.558678 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.558680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.558721 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:03:01.558822 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.558823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.558833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.558940 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.559100 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.559606 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.560652 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.561036 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.561127 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.561138 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.561289 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.561327 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:03:01.561408 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.558605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:44111 (local address 127.17.184.195:37119)
0927 04:03:01.558667 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.558669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.558709 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:03:01.558833 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.558834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.558850 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.558941 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.559103 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.559596 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.560649 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.561042 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.561133 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.561140 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.561304 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.561337 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 04:03:01.561430 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.558499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:37119 (local address 127.17.184.194:44111)
0927 04:03:01.558644 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.558657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.558729 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.558961 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.558962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.558964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.559031 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.559033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.559686 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.559687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.560551 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.560554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.561133 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.561134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.561218 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.561222 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.561286 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.561335 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":46,"thread_start_us":68,"threads_started":1}
I20260927 04:03:01.561430 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.558534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:42299 (local address 127.17.184.194:47825)
0927 04:03:01.558650 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.558661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.558738 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.558940 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.558941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.558943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.559026 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.559028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.559764 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.559770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.560553 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.560555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.561126 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.561127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.561211 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.561214 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.561284 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.561332 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":23,"thread_start_us":56,"threads_started":1}
I20260927 04:03:01.561830 20025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f3df6c23da4f3382119215411d6ddf" candidate_uuid: "e18ae88d43f44d76a99c837f450fe0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c37a3c630045109e15abf6ee8a59a3" is_pre_election: true
I20260927 04:03:01.561937 20025 raft_consensus.cc:2454] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18ae88d43f44d76a99c837f450fe0bb in term 0.
I20260927 04:03:01.561964 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f3df6c23da4f3382119215411d6ddf" candidate_uuid: "e18ae88d43f44d76a99c837f450fe0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" is_pre_election: true
I20260927 04:03:01.562100 20308 raft_consensus.cc:2454] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18ae88d43f44d76a99c837f450fe0bb in term 0.
I20260927 04:03:01.562147 20102 leader_election.cc:302] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e18ae88d43f44d76a99c837f450fe0bb, e3c37a3c630045109e15abf6ee8a59a3; no voters:
I20260927 04:03:01.562314 20383 raft_consensus.cc:2781] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:01.562350 20383 raft_consensus.cc:482] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:01.562376 20383 raft_consensus.cc:3037] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.563133 20383 raft_consensus.cc:504] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.563354 20383 leader_election.cc:288] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [CANDIDATE]: Term 1 election: Requested vote from peers e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299), 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:01.563488 20025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f3df6c23da4f3382119215411d6ddf" candidate_uuid: "e18ae88d43f44d76a99c837f450fe0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c37a3c630045109e15abf6ee8a59a3"
I20260927 04:03:01.563586 20025 raft_consensus.cc:3037] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.563570 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f3df6c23da4f3382119215411d6ddf" candidate_uuid: "e18ae88d43f44d76a99c837f450fe0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c99b0dda24e049ad95d4b8c6bca5e"
I20260927 04:03:01.563637 20308 raft_consensus.cc:3037] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.564435 20308 raft_consensus.cc:2454] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18ae88d43f44d76a99c837f450fe0bb in term 1.
I20260927 04:03:01.564435 20025 raft_consensus.cc:2454] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18ae88d43f44d76a99c837f450fe0bb in term 1.
I20260927 04:03:01.564628 20101 leader_election.cc:302] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206c99b0dda24e049ad95d4b8c6bca5e, e18ae88d43f44d76a99c837f450fe0bb; no voters:
I20260927 04:03:01.564792 20383 raft_consensus.cc:2781] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:01.564940 20383 raft_consensus.cc:686] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [term 1 LEADER]: Becoming Leader. State: Replica: e18ae88d43f44d76a99c837f450fe0bb, State: Running, Role: LEADER
I20260927 04:03:01.565060 20383 consensus_queue.cc:237] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.565481 20400 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: TxnStatusTablet state changed. Reason: New leader e18ae88d43f44d76a99c837f450fe0bb. Latest consensus state: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.565572 20400 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:03:01.565809 20402 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 04:03:01.565938 20402 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 04:03:01.565896 19869 catalog_manager.cc:5495] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb reported cstate change: term changed from 0 to 1, leader changed from <none> to e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194). New cstate: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:01.623876 20382 consensus_queue.cc:1035] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:01.626056 20387 consensus_queue.cc:1035] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:01.632011 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.628851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.17.184.194:35469)
0927 04:03:01.628902 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.628904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.629103 (+ 199us) server_negotiation.cc:407] Connection header received
0927 04:03:01.629227 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.629229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.629250 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.629371 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:01.629512 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.629995 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.631071 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.631265 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.631326 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.631566 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.631658 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.631721 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.631921 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.631944 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:03:01.632061 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.628976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35469 (local address 127.0.0.1:58100)
0927 04:03:01.629040 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.629054 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.629123 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.629377 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.629378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.629379 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:01.629435 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.629438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.630085 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.630087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.630967 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.630971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.631355 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.631356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.631428 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.631447 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.631819 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.631829 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.631920 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.631974 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 04:03:01.633690 20025 raft_consensus.cc:1261] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Refusing update from remote peer e18ae88d43f44d76a99c837f450fe0bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:01.633785 20308 raft_consensus.cc:1261] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Refusing update from remote peer e18ae88d43f44d76a99c837f450fe0bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:01.633961 20383 consensus_queue.cc:1035] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:01.634060 20400 consensus_queue.cc:1035] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:01.635079 20387 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: TxnStatusTablet state changed. Reason: New leader e18ae88d43f44d76a99c837f450fe0bb. Latest consensus state: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.635067 20385 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: TxnStatusTablet state changed. Reason: New leader e18ae88d43f44d76a99c837f450fe0bb. Latest consensus state: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.635154 20387 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:01.635154 20385 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:01.636094 20400 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.636190 20400 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:03:01.636261 20406 mvcc.cc:205] Tried to move back new op lower bound from 7333813377569411072 to 7333813377290997760. Current Snapshot: MvccSnapshot[applied={T|T < 7333813377569411072}]
I20260927 04:03:01.636955 20387 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.637063 20387 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P 206c99b0dda24e049ad95d4b8c6bca5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:01.637717 20400 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.637832 20400 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P e18ae88d43f44d76a99c837f450fe0bb: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:03:01.638652 20385 tablet_replica.cc:439] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } } }
I20260927 04:03:01.638767 20385 tablet_replica.cc:442] T d6f3df6c23da4f3382119215411d6ddf P e3c37a3c630045109e15abf6ee8a59a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:01.642300 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.639450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.17.184.195:37119)
0927 04:03:01.639518 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.639521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.639666 (+ 145us) server_negotiation.cc:407] Connection header received
0927 04:03:01.639710 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.639711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.639731 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.639817 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:01.639904 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.640600 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.641393 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.641689 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.641782 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.641947 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.642073 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.642167 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.642198 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.642223 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 04:03:01.648108 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.644911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:49017 (local address 127.17.184.194:35469)
0927 04:03:01.644979 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.644982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.645165 (+ 183us) server_negotiation.cc:407] Connection header received
0927 04:03:01.645248 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.645249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.645261 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.645332 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:01.645544 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.646247 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.647385 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.647571 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.647631 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.647870 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.647897 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.647935 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.648026 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.648046 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:03:01.648178 20420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.644864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35469 (local address 127.17.184.195:49017)
0927 04:03:01.645061 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.645074 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.645170 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.645356 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.645357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.645359 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:01.645432 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.645434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.646370 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.646371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.647289 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.647293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.647661 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.647662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.647735 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.647756 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.647960 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.647962 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.648027 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.648076 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":105,"threads_started":1}
I20260927 04:03:01.659634 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.656398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:37119 (local address 127.17.184.194:47935)
0927 04:03:01.656579 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.656589 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.656633 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.656786 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.656786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.656787 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:01.656847 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.656848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.657726 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.657727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.658718 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.658721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.659216 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.659217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.659268 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.659280 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.659479 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.659481 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.659513 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.659550 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":109,"threads_started":1}
I20260927 04:03:01.659642 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.656428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:47935 (local address 127.17.184.195:37119)
0927 04:03:01.656649 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.656652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.656663 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:01.656689 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.656690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.656705 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.656784 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:01.656897 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.657618 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.658802 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.659127 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.659215 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.659372 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.659411 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.659475 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.659533 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.659557 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 04:03:01.668804 19871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:01.668970 19871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:01.670037 19871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: Sending DeleteTablet for 3 replicas of tablet 1c98e3b89a404b47867197cdf021a1f3
I20260927 04:03:01.670444 20148 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c98e3b89a404b47867197cdf021a1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:01 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:58094
I20260927 04:03:01.670545 20288 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c98e3b89a404b47867197cdf021a1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:01 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41248
I20260927 04:03:01.670764 20008 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c98e3b89a404b47867197cdf021a1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:01 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:52884
I20260927 04:03:01.671058 20426 tablet_replica.cc:330] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: stopping tablet replica
I20260927 04:03:01.671177 20426 raft_consensus.cc:2229] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:01.671242 20426 raft_consensus.cc:2258] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:01.671219 19871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:01.671375 19871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:01.671406 20428 tablet_replica.cc:330] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: stopping tablet replica
I20260927 04:03:01.671507 20428 raft_consensus.cc:2229] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:01.671563 20428 raft_consensus.cc:2258] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:01.672015 20426 ts_tablet_manager.cc:1905] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:01.673038 20008 tablet_service.cc:1467] Processing CreateTablet for tablet a9bfb68708df4d14865559885f1ce7bc (DEFAULT_TABLE table=test-table [id=940e43bc3515482f97cc02d8e895209b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:01.673177 20008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bfb68708df4d14865559885f1ce7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.673995 20426 ts_tablet_manager.cc:1918] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:01.674063 20426 log.cc:1196] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/1c98e3b89a404b47867197cdf021a1f3
I20260927 04:03:01.674330 20426 ts_tablet_manager.cc:1939] T 1c98e3b89a404b47867197cdf021a1f3 P e18ae88d43f44d76a99c837f450fe0bb: Deleting consensus metadata
I20260927 04:03:01.674566 20381 tablet_bootstrap.cc:492] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: Bootstrap starting.
I20260927 04:03:01.675036 20381 tablet_bootstrap.cc:654] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.675175 19856 catalog_manager.cc:4841] TS e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469): tablet 1c98e3b89a404b47867197cdf021a1f3 (table test-table [id=85af126bd1f14f58b796da4b7982aad5]) successfully deleted
I20260927 04:03:01.675364 20148 tablet_service.cc:1467] Processing CreateTablet for tablet a9bfb68708df4d14865559885f1ce7bc (DEFAULT_TABLE table=test-table [id=940e43bc3515482f97cc02d8e895209b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:01.675465 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bfb68708df4d14865559885f1ce7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.675531 20288 tablet_service.cc:1467] Processing CreateTablet for tablet a9bfb68708df4d14865559885f1ce7bc (DEFAULT_TABLE table=test-table [id=940e43bc3515482f97cc02d8e895209b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:01.675637 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bfb68708df4d14865559885f1ce7bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:01.677274 20378 tablet_bootstrap.cc:492] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: Bootstrap starting.
I20260927 04:03:01.677313 20377 tablet_bootstrap.cc:492] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: Bootstrap starting.
I20260927 04:03:01.677728 20378 tablet_bootstrap.cc:654] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.677733 20377 tablet_bootstrap.cc:654] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:01.678696 20428 ts_tablet_manager.cc:1905] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:01.678784 20378 tablet_bootstrap.cc:492] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: No bootstrap required, opened a new log
I20260927 04:03:01.678839 20378 ts_tablet_manager.cc:1397] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:01.679014 20378 raft_consensus.cc:348] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.679095 20378 raft_consensus.cc:374] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.679144 20378 raft_consensus.cc:729] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18ae88d43f44d76a99c837f450fe0bb, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.679183 20378 consensus_queue.cc:260] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.679291 20378 ts_tablet_manager.cc:1428] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:01.679983 20377 tablet_bootstrap.cc:492] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: No bootstrap required, opened a new log
I20260927 04:03:01.680032 20377 ts_tablet_manager.cc:1397] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 04:03:01.680241 20377 raft_consensus.cc:348] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.680325 20377 raft_consensus.cc:374] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.680343 20377 raft_consensus.cc:729] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206c99b0dda24e049ad95d4b8c6bca5e, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.680387 20377 consensus_queue.cc:260] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.680486 20377 ts_tablet_manager.cc:1428] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:01.680896 20381 tablet_bootstrap.cc:492] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: No bootstrap required, opened a new log
I20260927 04:03:01.680959 20381 ts_tablet_manager.cc:1397] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260927 04:03:01.681130 20381 raft_consensus.cc:348] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.681242 20381 raft_consensus.cc:374] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:01.681270 20381 raft_consensus.cc:729] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c37a3c630045109e15abf6ee8a59a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:01.681313 20381 consensus_queue.cc:260] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.681401 20381 ts_tablet_manager.cc:1428] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:01.682047 20428 ts_tablet_manager.cc:1918] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:01.682127 20428 log.cc:1196] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/1c98e3b89a404b47867197cdf021a1f3
I20260927 04:03:01.682417 20428 ts_tablet_manager.cc:1939] T 1c98e3b89a404b47867197cdf021a1f3 P e3c37a3c630045109e15abf6ee8a59a3: Deleting consensus metadata
I20260927 04:03:01.682864 20427 tablet_replica.cc:330] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: stopping tablet replica
I20260927 04:03:01.682925 19858 catalog_manager.cc:4841] TS e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299): tablet 1c98e3b89a404b47867197cdf021a1f3 (table test-table [id=85af126bd1f14f58b796da4b7982aad5]) successfully deleted
I20260927 04:03:01.683013 20427 raft_consensus.cc:2229] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:01.683137 20427 raft_consensus.cc:2258] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:01.683552 20427 ts_tablet_manager.cc:1905] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:01.685289 20427 ts_tablet_manager.cc:1918] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:01.685374 20427 log.cc:1196] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/1c98e3b89a404b47867197cdf021a1f3
I20260927 04:03:01.685709 20427 ts_tablet_manager.cc:1939] T 1c98e3b89a404b47867197cdf021a1f3 P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting consensus metadata
I20260927 04:03:01.686209 19857 catalog_manager.cc:4841] TS 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119): tablet 1c98e3b89a404b47867197cdf021a1f3 (table test-table [id=85af126bd1f14f58b796da4b7982aad5]) successfully deleted
I20260927 04:03:01.827557 20385 raft_consensus.cc:482] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:01.827672 20385 raft_consensus.cc:504] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.828020 20385 leader_election.cc:288] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469), 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:01.831182 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.828311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35469 (local address 127.17.184.193:53281)
0927 04:03:01.828473 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.828483 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.828546 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.828739 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.828740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.828742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.828802 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.828803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.829459 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.829460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.830291 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.830293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.830842 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.830843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.830964 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.830969 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.831026 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.831071 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260927 04:03:01.831231 20375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.828354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:50533 (local address 127.17.184.195:37119)
0927 04:03:01.828420 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.828423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.828481 (+ 58us) server_negotiation.cc:407] Connection header received
0927 04:03:01.828572 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.828573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.828583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.828671 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.828817 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.829354 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.830435 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.830843 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.830982 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.830989 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.831127 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.831163 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 04:03:01.831559 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.828237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:37119 (local address 127.17.184.193:50533)
0927 04:03:01.828397 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.828410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.828490 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.828667 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.828668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.828671 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:01.828742 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.828744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.829494 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.829495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.830333 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.830335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.830976 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.830977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.831055 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.831058 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.831111 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.831258 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":14,"thread_start_us":91,"threads_started":1}
I20260927 04:03:01.831573 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.828437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:53281 (local address 127.17.184.194:35469)
0927 04:03:01.828504 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.828506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.828535 (+ 29us) server_negotiation.cc:407] Connection header received
0927 04:03:01.828616 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.828626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.828635 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.828681 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:01.828880 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.829370 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.830392 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.830749 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.830833 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.830843 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.831047 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.831108 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 04:03:01.831848 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bfb68708df4d14865559885f1ce7bc" candidate_uuid: "e3c37a3c630045109e15abf6ee8a59a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e18ae88d43f44d76a99c837f450fe0bb" is_pre_election: true
I20260927 04:03:01.831848 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bfb68708df4d14865559885f1ce7bc" candidate_uuid: "e3c37a3c630045109e15abf6ee8a59a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" is_pre_election: true
I20260927 04:03:01.831952 20308 raft_consensus.cc:2454] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c37a3c630045109e15abf6ee8a59a3 in term 0.
I20260927 04:03:01.831952 20168 raft_consensus.cc:2454] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c37a3c630045109e15abf6ee8a59a3 in term 0.
I20260927 04:03:01.832149 19961 leader_election.cc:302] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206c99b0dda24e049ad95d4b8c6bca5e, e3c37a3c630045109e15abf6ee8a59a3; no voters:
I20260927 04:03:01.832347 20385 raft_consensus.cc:2781] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:01.832402 20385 raft_consensus.cc:482] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:01.832437 20385 raft_consensus.cc:3037] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.833220 20385 raft_consensus.cc:504] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.833385 20385 leader_election.cc:288] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [CANDIDATE]: Term 1 election: Requested vote from peers e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469), 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:01.833541 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bfb68708df4d14865559885f1ce7bc" candidate_uuid: "e3c37a3c630045109e15abf6ee8a59a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c99b0dda24e049ad95d4b8c6bca5e"
I20260927 04:03:01.833542 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bfb68708df4d14865559885f1ce7bc" candidate_uuid: "e3c37a3c630045109e15abf6ee8a59a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e18ae88d43f44d76a99c837f450fe0bb"
I20260927 04:03:01.833623 20308 raft_consensus.cc:3037] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.833648 20168 raft_consensus.cc:3037] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:01.834301 20168 raft_consensus.cc:2454] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c37a3c630045109e15abf6ee8a59a3 in term 1.
I20260927 04:03:01.834306 20308 raft_consensus.cc:2454] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c37a3c630045109e15abf6ee8a59a3 in term 1.
I20260927 04:03:01.834496 19960 leader_election.cc:302] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e18ae88d43f44d76a99c837f450fe0bb, e3c37a3c630045109e15abf6ee8a59a3; no voters:
I20260927 04:03:01.834607 20385 raft_consensus.cc:2781] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:01.834774 20385 raft_consensus.cc:686] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 1 LEADER]: Becoming Leader. State: Replica: e3c37a3c630045109e15abf6ee8a59a3, State: Running, Role: LEADER
I20260927 04:03:01.834837 20385 consensus_queue.cc:237] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:01.835731 19871 catalog_manager.cc:5495] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "e3c37a3c630045109e15abf6ee8a59a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:01.868129 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.865392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.17.184.193:42299)
0927 04:03:01.865478 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.865480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.865490 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:01.865539 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.865540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.865563 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.865698 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:01.865915 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.866402 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.867301 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.867508 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.867573 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.867747 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.867863 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.867924 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.867976 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.867996 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 04:03:01.873823 20376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.871376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:39655 (local address 127.17.184.194:35469)
0927 04:03:01.871443 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.871445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.871517 (+ 72us) server_negotiation.cc:407] Connection header received
0927 04:03:01.871571 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.871571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.871580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.871635 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:01.871778 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.872268 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.873089 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.873278 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.873339 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.873579 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.873611 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.873667 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.873743 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.873762 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:03:01.873867 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.871283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35469 (local address 127.17.184.193:39655)
0927 04:03:01.871464 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.871474 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.871527 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.871647 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.871648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.871649 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:01.871716 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.871717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.872372 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.872373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.872998 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.873000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.873389 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.873390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.873446 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.873460 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.873695 (+ 235us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.873698 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.873740 (+ 42us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.873779 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260927 04:03:01.876049 20168 raft_consensus.cc:1261] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Refusing update from remote peer e3c37a3c630045109e15abf6ee8a59a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:01.876049 20308 raft_consensus.cc:1261] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Refusing update from remote peer e3c37a3c630045109e15abf6ee8a59a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:01.876354 20434 consensus_queue.cc:1035] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:01.876513 20385 consensus_queue.cc:1035] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:01.877775 20408 mvcc.cc:205] Tried to move back new op lower bound from 7333813378562764800 to 7333813378396413952. Current Snapshot: MvccSnapshot[applied={T|T < 7333813378562764800}]
I20260927 04:03:01.884351 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.881705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:41103 (local address 127.17.184.193:42299)
0927 04:03:01.881756 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:01.881758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:01.881766 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:01.881827 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:01.881828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:01.881836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:01.881912 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:01.882028 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.882640 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.883513 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.883842 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.883901 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.884039 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.884073 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.884137 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:01.884264 (+ 127us) server_negotiation.cc:299] Negotiation successful
0927 04:03:01.884285 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 04:03:01.884359 20422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:01.881636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:42299 (local address 127.17.184.194:41103)
0927 04:03:01.881700 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:01.881712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:01.881760 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:01.881912 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:01.881912 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:01.881914 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:01.881961 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.881962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.882728 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.882729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.883414 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:01.883417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:01.883899 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:01.883900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:01.883953 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:01.883959 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:01.884186 (+ 227us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:01.884189 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:01.884252 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 04:03:01.884302 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 04:03:01.884742 20408 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 04:03:03.409910 19869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:03.410017 19869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:03.410877 19869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: Sending DeleteTablet for 3 replicas of tablet a9bfb68708df4d14865559885f1ce7bc
I20260927 04:03:03.411152 20008 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bfb68708df4d14865559885f1ce7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:03 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:52884
I20260927 04:03:03.411190 20148 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bfb68708df4d14865559885f1ce7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:03 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:58094
I20260927 04:03:03.411357 20464 tablet_replica.cc:330] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: stopping tablet replica
I20260927 04:03:03.411430 20465 tablet_replica.cc:330] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: stopping tablet replica
I20260927 04:03:03.411435 20464 raft_consensus.cc:2229] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:03.411419 20288 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bfb68708df4d14865559885f1ce7bc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:03 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41248
I20260927 04:03:03.411494 20465 raft_consensus.cc:2229] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:03.411544 20464 raft_consensus.cc:2258] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:03.411582 20465 raft_consensus.cc:2258] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:03.411882 20465 ts_tablet_manager.cc:1905] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:03.411882 20464 ts_tablet_manager.cc:1905] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:03.413543 20466 tablet_replica.cc:330] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: stopping tablet replica
I20260927 04:03:03.413655 20466 raft_consensus.cc:2229] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:03.413709 20466 raft_consensus.cc:2258] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:03.413858 20465 ts_tablet_manager.cc:1918] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:03:03.413954 20465 log.cc:1196] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/a9bfb68708df4d14865559885f1ce7bc
I20260927 04:03:03.413992 20466 ts_tablet_manager.cc:1905] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:03.414263 20464 ts_tablet_manager.cc:1918] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:03:03.414294 20465 ts_tablet_manager.cc:1939] T a9bfb68708df4d14865559885f1ce7bc P e18ae88d43f44d76a99c837f450fe0bb: Deleting consensus metadata
I20260927 04:03:03.414392 20464 log.cc:1196] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/a9bfb68708df4d14865559885f1ce7bc
I20260927 04:03:03.414615 19856 catalog_manager.cc:4841] TS e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469): tablet a9bfb68708df4d14865559885f1ce7bc (table test-table [id=940e43bc3515482f97cc02d8e895209b]) successfully deleted
I20260927 04:03:03.414829 20464 ts_tablet_manager.cc:1939] T a9bfb68708df4d14865559885f1ce7bc P e3c37a3c630045109e15abf6ee8a59a3: Deleting consensus metadata
I20260927 04:03:03.415223 19858 catalog_manager.cc:4841] TS e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299): tablet a9bfb68708df4d14865559885f1ce7bc (table test-table [id=940e43bc3515482f97cc02d8e895209b]) successfully deleted
I20260927 04:03:03.415709 20466 ts_tablet_manager.cc:1918] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:03:03.415793 20466 log.cc:1196] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/a9bfb68708df4d14865559885f1ce7bc
I20260927 04:03:03.416045 20466 ts_tablet_manager.cc:1939] T a9bfb68708df4d14865559885f1ce7bc P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting consensus metadata
I20260927 04:03:03.416488 19857 catalog_manager.cc:4841] TS 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119): tablet a9bfb68708df4d14865559885f1ce7bc (table test-table [id=940e43bc3515482f97cc02d8e895209b]) successfully deleted
Sep 27 04:03:03 dist-test-slave-zqrn krb5kdc[19826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:03:03 dist-test-slave-zqrn krb5kdc[19826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481783, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:03.442864 20474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:03.428840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.17.184.254:39611)
0927 04:03:03.429011 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:03.429013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:03.429025 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:03:03.429050 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:03.429051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:03.429062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:03.429159 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:03.429737 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.430325 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.431138 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.431370 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.431439 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:03.440010 (+ 8571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:03.440012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:03.440012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:03.440022 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:03.441620 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:03.442068 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:03.442070 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:03.442070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:03.442138 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:03.442377 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:03.442378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:03.442378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:03.442497 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:03.442577 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:03.442676 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 04:03:03.442753 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260927 04:03:03.444614 19869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:03.444763 19869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:03.446483 20288 tablet_service.cc:1467] Processing CreateTablet for tablet 9eae50804d6c48568ede8989fb61794e (DEFAULT_TABLE table=test-table [id=e02949c791464ad48cf728f6a9f47d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:03.446502 20008 tablet_service.cc:1467] Processing CreateTablet for tablet 9eae50804d6c48568ede8989fb61794e (DEFAULT_TABLE table=test-table [id=e02949c791464ad48cf728f6a9f47d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:03.446667 20008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eae50804d6c48568ede8989fb61794e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:03.446664 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eae50804d6c48568ede8989fb61794e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:03.447901 20475 tablet_bootstrap.cc:492] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: Bootstrap starting.
I20260927 04:03:03.448001 20476 tablet_bootstrap.cc:492] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: Bootstrap starting.
I20260927 04:03:03.448065 20148 tablet_service.cc:1467] Processing CreateTablet for tablet 9eae50804d6c48568ede8989fb61794e (DEFAULT_TABLE table=test-table [id=e02949c791464ad48cf728f6a9f47d0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:03.448225 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eae50804d6c48568ede8989fb61794e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:03.448313 20475 tablet_bootstrap.cc:654] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:03.448382 20476 tablet_bootstrap.cc:654] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:03.448953 20476 tablet_bootstrap.cc:492] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: No bootstrap required, opened a new log
I20260927 04:03:03.449024 20476 ts_tablet_manager.cc:1397] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:03.449142 20476 raft_consensus.cc:348] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.449211 20476 raft_consensus.cc:374] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:03.449225 20476 raft_consensus.cc:729] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206c99b0dda24e049ad95d4b8c6bca5e, State: Initialized, Role: FOLLOWER
I20260927 04:03:03.449301 20476 consensus_queue.cc:260] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.449472 20476 ts_tablet_manager.cc:1428] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:03.449780 20480 tablet_bootstrap.cc:492] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: Bootstrap starting.
I20260927 04:03:03.450186 20480 tablet_bootstrap.cc:654] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:03.450248 20475 tablet_bootstrap.cc:492] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: No bootstrap required, opened a new log
I20260927 04:03:03.450330 20475 ts_tablet_manager.cc:1397] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:03.450497 20475 raft_consensus.cc:348] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.450630 20475 raft_consensus.cc:374] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:03.450671 20475 raft_consensus.cc:729] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c37a3c630045109e15abf6ee8a59a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:03.450752 20475 consensus_queue.cc:260] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.450853 20480 tablet_bootstrap.cc:492] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: No bootstrap required, opened a new log
I20260927 04:03:03.450881 20475 ts_tablet_manager.cc:1428] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:03.450942 20480 ts_tablet_manager.cc:1397] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:03.451069 20480 raft_consensus.cc:348] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.451148 20480 raft_consensus.cc:374] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:03.451189 20480 raft_consensus.cc:729] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18ae88d43f44d76a99c837f450fe0bb, State: Initialized, Role: FOLLOWER
I20260927 04:03:03.451270 20480 consensus_queue.cc:260] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.451404 20480 ts_tablet_manager.cc:1428] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:03.496670 20450 raft_consensus.cc:482] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:03.496776 20450 raft_consensus.cc:504] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.496948 20450 leader_election.cc:288] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299), 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:03.497188 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eae50804d6c48568ede8989fb61794e" candidate_uuid: "e18ae88d43f44d76a99c837f450fe0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" is_pre_election: true
I20260927 04:03:03.497156 20025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eae50804d6c48568ede8989fb61794e" candidate_uuid: "e18ae88d43f44d76a99c837f450fe0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c37a3c630045109e15abf6ee8a59a3" is_pre_election: true
I20260927 04:03:03.497305 20308 raft_consensus.cc:2454] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18ae88d43f44d76a99c837f450fe0bb in term 0.
I20260927 04:03:03.497299 20025 raft_consensus.cc:2454] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e18ae88d43f44d76a99c837f450fe0bb in term 0.
I20260927 04:03:03.497494 20101 leader_election.cc:302] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206c99b0dda24e049ad95d4b8c6bca5e, e18ae88d43f44d76a99c837f450fe0bb; no voters:
I20260927 04:03:03.497622 20450 raft_consensus.cc:2781] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:03.497671 20450 raft_consensus.cc:482] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:03.497699 20450 raft_consensus.cc:3037] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:03.498457 20450 raft_consensus.cc:504] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.498733 20450 leader_election.cc:288] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [CANDIDATE]: Term 1 election: Requested vote from peers e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299), 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:03.498937 20025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eae50804d6c48568ede8989fb61794e" candidate_uuid: "e18ae88d43f44d76a99c837f450fe0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3c37a3c630045109e15abf6ee8a59a3"
I20260927 04:03:03.498973 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eae50804d6c48568ede8989fb61794e" candidate_uuid: "e18ae88d43f44d76a99c837f450fe0bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c99b0dda24e049ad95d4b8c6bca5e"
I20260927 04:03:03.499047 20025 raft_consensus.cc:3037] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:03.499049 20308 raft_consensus.cc:3037] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:03.499686 20308 raft_consensus.cc:2454] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18ae88d43f44d76a99c837f450fe0bb in term 1.
I20260927 04:03:03.499706 20025 raft_consensus.cc:2454] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e18ae88d43f44d76a99c837f450fe0bb in term 1.
I20260927 04:03:03.499884 20101 leader_election.cc:302] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 206c99b0dda24e049ad95d4b8c6bca5e, e18ae88d43f44d76a99c837f450fe0bb; no voters:
I20260927 04:03:03.499979 20450 raft_consensus.cc:2781] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:03.500090 20450 raft_consensus.cc:686] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 1 LEADER]: Becoming Leader. State: Replica: e18ae88d43f44d76a99c837f450fe0bb, State: Running, Role: LEADER
I20260927 04:03:03.500150 20450 consensus_queue.cc:237] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.500975 19871 catalog_manager.cc:5495] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb reported cstate change: term changed from 0 to 1, leader changed from <none> to e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194). New cstate: current_term: 1 leader_uuid: "e18ae88d43f44d76a99c837f450fe0bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:03.513952 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:03.511444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58102 (local address 127.17.184.194:35469)
0927 04:03:03.511598 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:03.511599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:03.511605 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:03:03.511624 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:03.511624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:03.511633 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:03.511707 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:03.511880 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.512390 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.513192 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.513434 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.513511 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:03.513664 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:03.513698 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:03.513762 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:03.513831 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 04:03:03.513856 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260927 04:03:03.518573 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:03.516521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:47997 (local address 127.17.184.194:35469)
0927 04:03:03.516580 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:03.516582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:03.516589 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:03:03.516611 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:03.516612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:03.516621 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:03.516674 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:03.516787 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.517214 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.517945 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.518125 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.518174 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:03.518348 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:03.518383 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:03.518421 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:03.518475 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 04:03:03.518494 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 04:03:03.518589 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:03.516365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35469 (local address 127.17.184.194:47997)
0927 04:03:03.516500 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:03.516510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:03.516552 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:03.516689 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:03.516690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:03.516692 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:03.516730 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:03.516731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.517299 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.517300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:03.517871 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:03.517873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.518227 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.518228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:03.518275 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:03.518279 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:03.518438 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:03.518439 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 04:03:03.518474 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 04:03:03.518513 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":87,"threads_started":1}
I20260927 04:03:03.520885 20025 raft_consensus.cc:1261] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Refusing update from remote peer e18ae88d43f44d76a99c837f450fe0bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:03.520891 20308 raft_consensus.cc:1261] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Refusing update from remote peer e18ae88d43f44d76a99c837f450fe0bb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:03.521099 20445 consensus_queue.cc:1035] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:03.521234 20383 consensus_queue.cc:1035] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [LEADER]: Connected to new peer: Peer: permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:03.532215 19869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:03.532366 19869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:03.533051 19869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: Sending DeleteTablet for 3 replicas of tablet 9eae50804d6c48568ede8989fb61794e
I20260927 04:03:03.533346 20288 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eae50804d6c48568ede8989fb61794e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:03 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41248
I20260927 04:03:03.533383 20008 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eae50804d6c48568ede8989fb61794e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:03 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:52884
I20260927 04:03:03.533464 20466 tablet_replica.cc:330] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: stopping tablet replica
I20260927 04:03:03.533488 20464 tablet_replica.cc:330] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: stopping tablet replica
I20260927 04:03:03.533524 20466 raft_consensus.cc:2229] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:03.533543 20464 raft_consensus.cc:2229] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:03.533567 20466 raft_consensus.cc:2258] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:03.533578 20464 raft_consensus.cc:2258] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:03.533726 19869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:03.533867 19869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:03.533970 20464 ts_tablet_manager.cc:1905] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:03.534008 20466 ts_tablet_manager.cc:1905] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:03.535589 20464 ts_tablet_manager.cc:1918] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:03.535662 20464 log.cc:1196] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/9eae50804d6c48568ede8989fb61794e
I20260927 04:03:03.535655 20148 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eae50804d6c48568ede8989fb61794e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:03 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:58094
I20260927 04:03:03.535755 20465 tablet_replica.cc:330] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: stopping tablet replica
I20260927 04:03:03.535816 20465 raft_consensus.cc:2229] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:03.535916 20464 ts_tablet_manager.cc:1939] T 9eae50804d6c48568ede8989fb61794e P e3c37a3c630045109e15abf6ee8a59a3: Deleting consensus metadata
I20260927 04:03:03.535928 20465 raft_consensus.cc:2258] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:03.535935 20466 ts_tablet_manager.cc:1918] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:03.535984 20466 log.cc:1196] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/9eae50804d6c48568ede8989fb61794e
I20260927 04:03:03.536206 20466 ts_tablet_manager.cc:1939] T 9eae50804d6c48568ede8989fb61794e P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting consensus metadata
I20260927 04:03:03.536345 19858 catalog_manager.cc:4841] TS e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299): tablet 9eae50804d6c48568ede8989fb61794e (table test-table [id=e02949c791464ad48cf728f6a9f47d0f]) successfully deleted
I20260927 04:03:03.536593 19857 catalog_manager.cc:4841] TS 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119): tablet 9eae50804d6c48568ede8989fb61794e (table test-table [id=e02949c791464ad48cf728f6a9f47d0f]) successfully deleted
I20260927 04:03:03.536931 20288 tablet_service.cc:1467] Processing CreateTablet for tablet c6439e2d398744c599aeb7a762d72fc3 (DEFAULT_TABLE table=test-table [id=1512752da50b447d9402a0f014a3b7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:03.536938 20008 tablet_service.cc:1467] Processing CreateTablet for tablet c6439e2d398744c599aeb7a762d72fc3 (DEFAULT_TABLE table=test-table [id=1512752da50b447d9402a0f014a3b7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:03.537042 20008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6439e2d398744c599aeb7a762d72fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:03.537065 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6439e2d398744c599aeb7a762d72fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:03.538030 20476 tablet_bootstrap.cc:492] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: Bootstrap starting.
I20260927 04:03:03.538247 20475 tablet_bootstrap.cc:492] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3: Bootstrap starting.
I20260927 04:03:03.538405 20476 tablet_bootstrap.cc:654] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:03.538432 20465 ts_tablet_manager.cc:1905] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:03.538571 20475 tablet_bootstrap.cc:654] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:03.539186 20475 tablet_bootstrap.cc:492] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3: No bootstrap required, opened a new log
I20260927 04:03:03.539234 20475 ts_tablet_manager.cc:1397] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:03.539363 20475 raft_consensus.cc:348] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.539426 20476 tablet_bootstrap.cc:492] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: No bootstrap required, opened a new log
I20260927 04:03:03.539436 20475 raft_consensus.cc:374] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:03.539459 20475 raft_consensus.cc:729] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c37a3c630045109e15abf6ee8a59a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:03.539471 20476 ts_tablet_manager.cc:1397] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:03.539510 20475 consensus_queue.cc:260] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.539598 20475 ts_tablet_manager.cc:1428] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:03.539574 20476 raft_consensus.cc:348] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.539638 20476 raft_consensus.cc:374] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:03.539659 20476 raft_consensus.cc:729] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 206c99b0dda24e049ad95d4b8c6bca5e, State: Initialized, Role: FOLLOWER
I20260927 04:03:03.539700 20476 consensus_queue.cc:260] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.539813 20476 ts_tablet_manager.cc:1428] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:03.540084 20465 ts_tablet_manager.cc:1918] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:03.540141 20465 log.cc:1196] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/9eae50804d6c48568ede8989fb61794e
I20260927 04:03:03.540359 20465 ts_tablet_manager.cc:1939] T 9eae50804d6c48568ede8989fb61794e P e18ae88d43f44d76a99c837f450fe0bb: Deleting consensus metadata
I20260927 04:03:03.540912 20148 tablet_service.cc:1467] Processing CreateTablet for tablet c6439e2d398744c599aeb7a762d72fc3 (DEFAULT_TABLE table=test-table [id=1512752da50b447d9402a0f014a3b7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:03.541036 20148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6439e2d398744c599aeb7a762d72fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:03.541895 20480 tablet_bootstrap.cc:492] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: Bootstrap starting.
I20260927 04:03:03.542187 20480 tablet_bootstrap.cc:654] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:03.542502 19856 catalog_manager.cc:4841] TS e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469): tablet 9eae50804d6c48568ede8989fb61794e (table test-table [id=e02949c791464ad48cf728f6a9f47d0f]) successfully deleted
I20260927 04:03:03.543363 20480 tablet_bootstrap.cc:492] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: No bootstrap required, opened a new log
I20260927 04:03:03.543437 20480 ts_tablet_manager.cc:1397] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:03.543555 20480 raft_consensus.cc:348] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.543632 20480 raft_consensus.cc:374] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:03.543659 20480 raft_consensus.cc:729] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e18ae88d43f44d76a99c837f450fe0bb, State: Initialized, Role: FOLLOWER
I20260927 04:03:03.543699 20480 consensus_queue.cc:260] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.543798 20480 ts_tablet_manager.cc:1428] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:03.619113 20458 raft_consensus.cc:482] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:03.619248 20458 raft_consensus.cc:504] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.619432 20458 leader_election.cc:288] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469), 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:03.619623 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6439e2d398744c599aeb7a762d72fc3" candidate_uuid: "e3c37a3c630045109e15abf6ee8a59a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" is_pre_election: true
I20260927 04:03:03.619627 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6439e2d398744c599aeb7a762d72fc3" candidate_uuid: "e3c37a3c630045109e15abf6ee8a59a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e18ae88d43f44d76a99c837f450fe0bb" is_pre_election: true
I20260927 04:03:03.619733 20168 raft_consensus.cc:2454] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c37a3c630045109e15abf6ee8a59a3 in term 0.
I20260927 04:03:03.619752 20308 raft_consensus.cc:2454] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3c37a3c630045109e15abf6ee8a59a3 in term 0.
I20260927 04:03:03.619941 19960 leader_election.cc:302] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e18ae88d43f44d76a99c837f450fe0bb, e3c37a3c630045109e15abf6ee8a59a3; no voters:
I20260927 04:03:03.620067 20458 raft_consensus.cc:2781] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:03.620105 20458 raft_consensus.cc:482] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:03.620137 20458 raft_consensus.cc:3037] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:03.620870 20458 raft_consensus.cc:504] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.621018 20458 leader_election.cc:288] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [CANDIDATE]: Term 1 election: Requested vote from peers e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469), 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119)
I20260927 04:03:03.621152 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6439e2d398744c599aeb7a762d72fc3" candidate_uuid: "e3c37a3c630045109e15abf6ee8a59a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e18ae88d43f44d76a99c837f450fe0bb"
I20260927 04:03:03.621188 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6439e2d398744c599aeb7a762d72fc3" candidate_uuid: "e3c37a3c630045109e15abf6ee8a59a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "206c99b0dda24e049ad95d4b8c6bca5e"
I20260927 04:03:03.621265 20308 raft_consensus.cc:3037] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:03.621259 20168 raft_consensus.cc:3037] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:03.621901 20168 raft_consensus.cc:2454] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c37a3c630045109e15abf6ee8a59a3 in term 1.
I20260927 04:03:03.622066 20308 raft_consensus.cc:2454] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3c37a3c630045109e15abf6ee8a59a3 in term 1.
I20260927 04:03:03.622076 19960 leader_election.cc:302] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e18ae88d43f44d76a99c837f450fe0bb, e3c37a3c630045109e15abf6ee8a59a3; no voters:
I20260927 04:03:03.622150 20458 raft_consensus.cc:2781] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:03.622203 20458 raft_consensus.cc:686] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [term 1 LEADER]: Becoming Leader. State: Replica: e3c37a3c630045109e15abf6ee8a59a3, State: Running, Role: LEADER
I20260927 04:03:03.622259 20458 consensus_queue.cc:237] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } }
I20260927 04:03:03.622961 19871 catalog_manager.cc:5495] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "e3c37a3c630045109e15abf6ee8a59a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3c37a3c630045109e15abf6ee8a59a3" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 42299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:03.641263 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:03.638177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.17.184.193:42299)
0927 04:03:03.638378 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:03.638379 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:03.638385 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:03:03.638404 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:03.638404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:03.638415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:03.638477 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:03.638663 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.639265 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.640311 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:03.640529 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:03.640600 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:03.640873 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:03.640910 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:03.640965 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:03.641063 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 04:03:03.641096 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":120,"threads_started":1}
I20260927 04:03:03.644441 20168 raft_consensus.cc:1261] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Refusing update from remote peer e3c37a3c630045109e15abf6ee8a59a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:03.644444 20308 raft_consensus.cc:1261] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Refusing update from remote peer e3c37a3c630045109e15abf6ee8a59a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:03.644661 20434 consensus_queue.cc:1035] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "206c99b0dda24e049ad95d4b8c6bca5e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 37119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:03.644761 20458 consensus_queue.cc:1035] T c6439e2d398744c599aeb7a762d72fc3 P e3c37a3c630045109e15abf6ee8a59a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e18ae88d43f44d76a99c837f450fe0bb" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:03.646787 20446 mvcc.cc:205] Tried to move back new op lower bound from 7333813385806204928 to 7333813385717084160. Current Snapshot: MvccSnapshot[applied={T|T < 7333813385806204928}]
W20260927 04:03:03.649225 20448 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 04:03:05.167004 19871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:05.167145 19871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:05.168176 19871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb08e31c2d3847dd97a01bb54405e862: Sending DeleteTablet for 3 replicas of tablet c6439e2d398744c599aeb7a762d72fc3
I20260927 04:03:05.168428 20008 tablet_service.cc:1514] Processing DeleteTablet for tablet c6439e2d398744c599aeb7a762d72fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:05 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:52884
I20260927 04:03:05.168473 20147 tablet_service.cc:1514] Processing DeleteTablet for tablet c6439e2d398744c599aeb7a762d72fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:05 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:58094
I20260927 04:03:05.168613 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 19943
I20260927 04:03:05.168641 20288 tablet_service.cc:1514] Processing DeleteTablet for tablet c6439e2d398744c599aeb7a762d72fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:05 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41248
I20260927 04:03:05.169657 20520 tablet_replica.cc:330] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: stopping tablet replica
I20260927 04:03:05.169732 20520 raft_consensus.cc:2229] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:05.169777 20520 raft_consensus.cc:2258] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:05.170056 20520 ts_tablet_manager.cc:1905] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:05.170405 20521 tablet_replica.cc:330] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: stopping tablet replica
I20260927 04:03:05.170495 20521 raft_consensus.cc:2229] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:05.170547 20521 raft_consensus.cc:2258] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:05.170853 20521 ts_tablet_manager.cc:1905] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:05.172042 20520 ts_tablet_manager.cc:1918] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:03:05.172130 20520 log.cc:1196] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/c6439e2d398744c599aeb7a762d72fc3
I20260927 04:03:05.172377 20520 ts_tablet_manager.cc:1939] T c6439e2d398744c599aeb7a762d72fc3 P e18ae88d43f44d76a99c837f450fe0bb: Deleting consensus metadata
I20260927 04:03:05.172432 20521 ts_tablet_manager.cc:1918] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 04:03:05.172530 20521 log.cc:1196] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/c6439e2d398744c599aeb7a762d72fc3
I20260927 04:03:05.172760 20521 ts_tablet_manager.cc:1939] T c6439e2d398744c599aeb7a762d72fc3 P 206c99b0dda24e049ad95d4b8c6bca5e: Deleting consensus metadata
I20260927 04:03:05.172781 19856 catalog_manager.cc:4841] TS e18ae88d43f44d76a99c837f450fe0bb (127.17.184.194:35469): tablet c6439e2d398744c599aeb7a762d72fc3 (table test-table [id=1512752da50b447d9402a0f014a3b7a5]) successfully deleted
I20260927 04:03:05.173094 19857 catalog_manager.cc:4841] TS 206c99b0dda24e049ad95d4b8c6bca5e (127.17.184.195:37119): tablet c6439e2d398744c599aeb7a762d72fc3 (table test-table [id=1512752da50b447d9402a0f014a3b7a5]) successfully deleted
W20260927 04:03:05.177547 19858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:42299 (error 108)
I20260927 04:03:05.177915 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 20083
I20260927 04:03:05.183213 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:05.177988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:42299 (local address 127.0.0.1:52120)
0927 04:03:05.183042 (+ 5054us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:05.183155 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5006,"thread_start_us":92,"threads_started":1}
W20260927 04:03:05.183400 19858 catalog_manager.cc:4584] TS e3c37a3c630045109e15abf6ee8a59a3 (127.17.184.193:42299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6439e2d398744c599aeb7a762d72fc3: Network error: Client connection negotiation failed: client connection to 127.17.184.193:42299: connect: Connection refused (error 111)
I20260927 04:03:05.184741 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 20223
I20260927 04:03:05.191398 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 19839
2026-09-27T04:03:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5957 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:05 dist-test-slave-zqrn krb5kdc[20526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:05 dist-test-slave-zqrn krb5kdc[20526](info): set up 2 sockets
Sep 27 04:03:05 dist-test-slave-zqrn krb5kdc[20526](info): commencing operation
krb5kdc: starting...
W20260927 04:03:06.251091 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:06 dist-test-slave-zqrn krb5kdc[20526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:06.378594 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:43787
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:43787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:06.463537 20539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:06.463770 20539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:06.463793 20539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:06.464921 20539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:06.464962 20539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:06.464977 20539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:06.464988 20539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:06.465001 20539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:06.466426 20539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:43787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:43787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:06.466634 20539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:06.466806 20539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:03:06.472549 20539 server_base.cc:1034] running on GCE node
W20260927 04:03:06.472530 20547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:06.472530 20544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:06.472568 20545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:06.473140 20539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:06.473497 20539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:06.474678 20539 hybrid_clock.cc:648] HybridClock initialized: now 1790481786474661 us; error 43 us; skew 500 ppm
Sep 27 04:03:06 dist-test-slave-zqrn krb5kdc[20526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481786, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:06.477650 20539 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:06.478173 20539 webserver.cc:460] Webserver started at http://127.17.184.254:35979/ using document root <none> and password file <none>
I20260927 04:03:06.478391 20539 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:06.478437 20539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:06.478540 20539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:06.479815 20539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "0b2bbf39deb44672ad8c7bf2e37e0285"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "604a1fd5caf481a4fa2a593908ff2560"
server_key_iv: "7cfd5f166a5c1760075429ae57b5a37a"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.480157 20539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "0b2bbf39deb44672ad8c7bf2e37e0285"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "604a1fd5caf481a4fa2a593908ff2560"
server_key_iv: "7cfd5f166a5c1760075429ae57b5a37a"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.481593 20539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:06.482492 20554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:06.482712 20539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:06.482818 20539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "0b2bbf39deb44672ad8c7bf2e37e0285"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "604a1fd5caf481a4fa2a593908ff2560"
server_key_iv: "7cfd5f166a5c1760075429ae57b5a37a"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.482885 20539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:06.500422 20539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:06.502862 20539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:06.503054 20539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:06.506811 20539 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:43787
I20260927 04:03:06.506877 20606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:43787 every 8 connection(s)
I20260927 04:03:06.507265 20539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:06.508213 20607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:06.510542 20607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285: Bootstrap starting.
I20260927 04:03:06.511190 20607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:06.511492 20607 log.cc:824] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:06.512238 20607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285: No bootstrap required, opened a new log
I20260927 04:03:06.513902 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 20539
I20260927 04:03:06.514012 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:06.513921 20607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2bbf39deb44672ad8c7bf2e37e0285" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43787 } }
I20260927 04:03:06.514122 20607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:06.514196 20607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b2bbf39deb44672ad8c7bf2e37e0285, State: Initialized, Role: FOLLOWER
I20260927 04:03:06.514256 18147 external_mini_cluster.cc:1428] Setting key 4a6035ffe0deab8ed000731322d50f4a
I20260927 04:03:06.514384 20607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2bbf39deb44672ad8c7bf2e37e0285" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43787 } }
I20260927 04:03:06.514485 20607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:06.514523 20607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:06.514580 20607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:06.515743 20607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2bbf39deb44672ad8c7bf2e37e0285" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43787 } }
I20260927 04:03:06.515836 20607 leader_election.cc:302] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b2bbf39deb44672ad8c7bf2e37e0285; no voters:
I20260927 04:03:06.516036 20607 leader_election.cc:288] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:06.516065 20612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:06.516183 20612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [term 1 LEADER]: Becoming Leader. State: Replica: 0b2bbf39deb44672ad8c7bf2e37e0285, State: Running, Role: LEADER
I20260927 04:03:06.516269 20612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2bbf39deb44672ad8c7bf2e37e0285" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43787 } }
I20260927 04:03:06.516321 20607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:06.516672 20614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b2bbf39deb44672ad8c7bf2e37e0285. Latest consensus state: current_term: 1 leader_uuid: "0b2bbf39deb44672ad8c7bf2e37e0285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2bbf39deb44672ad8c7bf2e37e0285" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43787 } } }
I20260927 04:03:06.516685 20613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b2bbf39deb44672ad8c7bf2e37e0285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b2bbf39deb44672ad8c7bf2e37e0285" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43787 } } }
I20260927 04:03:06.516770 20614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:06.516801 20613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:06.517030 20618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:06.517575 20618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:06.519254 20618 catalog_manager.cc:1347] Generated new cluster ID: 7e3a96b3e6c344a896d205b3eb7d05e8
I20260927 04:03:06.519320 20618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:03:06 dist-test-slave-zqrn krb5kdc[20526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:06.525671 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:06.514731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.17.184.254:43787)
0927 04:03:06.514952 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:06.514964 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:06.515021 (+ 57us) server_negotiation.cc:407] Connection header received
0927 04:03:06.515185 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:06.515195 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:06.515267 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:06.515447 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:06.516147 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.517975 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.519065 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.519645 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.519786 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:06.522175 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:06.522179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:06.522180 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:06.522192 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:06.524023 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:06.524416 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:06.524420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:06.524420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:06.524538 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:06.524768 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:06.524769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:06.524769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:06.525031 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:06.525200 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:06.525300 (+ 100us) server_negotiation.cc:299] Negotiation successful
0927 04:03:06.525444 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":114,"threads_started":1}
I20260927 04:03:06.536260 20618 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:06.537017 20618 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:06.546152 20618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b2bbf39deb44672ad8c7bf2e37e0285: Generated new TSK 0
I20260927 04:03:06.546411 20618 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:03:06.578975 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:43787
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:06.662469 20633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:06.662719 20633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:06.662757 20633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:06.664077 20633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:06.664141 20633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:06.664187 20633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:06.665655 20633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:06.665934 20633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:06.666174 20633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:06.671504 20639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:06.671497 20638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:06.671497 20641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:06.671600 20633 server_base.cc:1034] running on GCE node
I20260927 04:03:06.671924 20633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:06.672183 20633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:06.673532 20633 hybrid_clock.cc:648] HybridClock initialized: now 1790481786673479 us; error 44 us; skew 500 ppm
Sep 27 04:03:06 dist-test-slave-zqrn krb5kdc[20526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481786, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:06.676227 20633 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:03:06.676683 20633 webserver.cc:460] Webserver started at http://127.17.184.193:39783/ using document root <none> and password file <none>
I20260927 04:03:06.676890 20633 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:06.676930 20633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:06.677021 20633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:06.678049 20633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "71034b2007db4a749b3f3d66a807deae"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "992487e52fe38436de22aab87a0ce2a1"
server_key_iv: "b4eedb1475897b826485146dd95f0782"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.678345 20633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "71034b2007db4a749b3f3d66a807deae"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "992487e52fe38436de22aab87a0ce2a1"
server_key_iv: "b4eedb1475897b826485146dd95f0782"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.679776 20633 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 04:03:06.680562 20648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:06.680757 20633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:06.680871 20633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "71034b2007db4a749b3f3d66a807deae"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "992487e52fe38436de22aab87a0ce2a1"
server_key_iv: "b4eedb1475897b826485146dd95f0782"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.680956 20633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:06.694398 20633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:06.696877 20633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:06.697042 20633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:06.697293 20633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:06.697598 20633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:06.697651 20633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:06.697696 20633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:06.697718 20633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:06.703594 20633 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:45477
I20260927 04:03:06.703827 20761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:45477 every 8 connection(s)
I20260927 04:03:06.704044 20633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:06.704300 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 20633
I20260927 04:03:06.704407 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:06.704610 18147 external_mini_cluster.cc:1428] Setting key b30eadcf05c9ae1cf40880925026c88b
Sep 27 04:03:06 dist-test-slave-zqrn krb5kdc[20526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481786, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:03:06.715276 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:06.704873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:39197 (local address 127.17.184.254:43787)
0927 04:03:06.704941 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:06.704944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:06.705193 (+ 249us) server_negotiation.cc:407] Connection header received
0927 04:03:06.705892 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:06.705893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:06.705913 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:06.705968 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:06.707317 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.708176 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.709275 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.709606 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.709686 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:06.712188 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:06.712190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:06.712190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:06.712199 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:06.713871 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:06.714254 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:06.714255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:06.714255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:06.714310 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:06.714553 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:06.714555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:06.714555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:06.714732 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:06.714829 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:06.715117 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 04:03:06.715199 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:03:06.715476 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:06.704865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:43787 (local address 127.17.184.193:39197)
0927 04:03:06.705044 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:06.705086 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:06.705741 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:06.706057 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:06.706062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:06.706958 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:06.707221 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:06.707227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.708280 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.708280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:06.709174 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:06.709177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.709694 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.709695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:06.709828 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:06.710399 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:06.710407 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:06.712091 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:06.713949 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:06.713955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:06.713956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:06.714183 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:06.714372 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:06.714373 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:06.714373 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:06.714471 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:06.714812 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:06.714817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:06.714991 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 04:03:06.715085 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 04:03:06.715211 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260927 04:03:06.715966 20762 heartbeater.cc:347] Connected to a master server at 127.17.184.254:43787
I20260927 04:03:06.716056 20762 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
I20260927 04:03:06.716272 20762 heartbeater.cc:511] Master 127.17.184.254:43787 requested a full tablet report, sending...
I20260927 04:03:06.716791 20571 ts_manager.cc:194] Registered new tserver with Master: 71034b2007db4a749b3f3d66a807deae (127.17.184.193:45477)
I20260927 04:03:06.717605 20571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:39197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:03:06.735972 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:43787
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:06.818109 20767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:06.818321 20767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:06.818344 20767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:06.819552 20767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:06.819602 20767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:06.819629 20767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:06.821059 20767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:06.821307 20767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:06.821543 20767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:06.826696 20773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:06.826797 20772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:06.826831 20775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:06.827154 20767 server_base.cc:1034] running on GCE node
I20260927 04:03:06.827423 20767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:06.827764 20767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:06.828958 20767 hybrid_clock.cc:648] HybridClock initialized: now 1790481786828947 us; error 36 us; skew 500 ppm
Sep 27 04:03:06 dist-test-slave-zqrn krb5kdc[20526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481786, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:06.832113 20767 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:03:06.832966 20767 webserver.cc:460] Webserver started at http://127.17.184.194:44557/ using document root <none> and password file <none>
I20260927 04:03:06.833261 20767 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:06.833315 20767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:06.833455 20767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:06.835222 20767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "804a8cbaf34748a79c5b1360875bdce8"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "e92c7e1735b03295682c65848ff471ce"
server_key_iv: "aadfe7e4e14959f4713ae55ca6eb7d6f"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.835718 20767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "804a8cbaf34748a79c5b1360875bdce8"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "e92c7e1735b03295682c65848ff471ce"
server_key_iv: "aadfe7e4e14959f4713ae55ca6eb7d6f"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.841171 20767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 04:03:06.842355 20782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:06.842590 20767 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:06.842717 20767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "804a8cbaf34748a79c5b1360875bdce8"
format_stamp: "Formatted at 2026-09-27 04:03:06 on dist-test-slave-zqrn"
server_key: "e92c7e1735b03295682c65848ff471ce"
server_key_iv: "aadfe7e4e14959f4713ae55ca6eb7d6f"
server_key_version: "encryptionkey@0"
I20260927 04:03:06.842828 20767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:06.866811 20767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:06.869159 20767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:06.869344 20767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:06.869611 20767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:06.869953 20767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:06.870008 20767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:06.870045 20767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:06.870090 20767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:06.876747 20767 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:34427
I20260927 04:03:06.876796 20895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:34427 every 8 connection(s)
I20260927 04:03:06.877171 20767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:06.881637 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 20767
I20260927 04:03:06.881793 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:06.881987 18147 external_mini_cluster.cc:1428] Setting key c306543d1f9a18bf42064faea5de5be4
Sep 27 04:03:06 dist-test-slave-zqrn krb5kdc[20526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481786, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:06.888315 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:06.878090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:55939 (local address 127.17.184.254:43787)
0927 04:03:06.878172 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:06.878175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:06.878393 (+ 218us) server_negotiation.cc:407] Connection header received
0927 04:03:06.879054 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:06.879055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:06.879067 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:06.879132 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:06.880110 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.880666 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.881715 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.882082 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.882170 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:06.884868 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:06.884870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:06.884870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:06.884877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:06.886591 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:06.887196 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:06.887197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:06.887198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:06.887252 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:06.887588 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:06.887589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:06.887589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:06.887706 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:06.887758 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:06.888166 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 04:03:06.888240 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 04:03:06.888696 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:06.878052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:43787 (local address 127.17.184.194:55939)
0927 04:03:06.878255 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:06.878296 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:06.878946 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:06.879183 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:06.879187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:06.879774 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:06.879991 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:06.879995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.880759 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.880760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:06.881600 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:06.881602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:06.882179 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:06.882180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:06.882350 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:06.882995 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:06.883005 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:06.884752 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:06.886672 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:06.886680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:06.886682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:06.887120 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:06.887325 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:06.887326 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:06.887327 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:06.887505 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:06.887775 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:06.887781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:06.887990 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 04:03:06.888141 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 04:03:06.888358 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260927 04:03:06.889285 20896 heartbeater.cc:347] Connected to a master server at 127.17.184.254:43787
I20260927 04:03:06.889408 20896 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:06.889595 20896 heartbeater.cc:511] Master 127.17.184.254:43787 requested a full tablet report, sending...
I20260927 04:03:06.890002 20571 ts_manager.cc:194] Registered new tserver with Master: 804a8cbaf34748a79c5b1360875bdce8 (127.17.184.194:34427)
I20260927 04:03:06.890404 20571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:55939
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:03:06.910403 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:43787
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:06.993803 20901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:06.994026 20901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:06.994052 20901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:06.995400 20901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:06.995442 20901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:06.995471 20901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:06.996953 20901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:43787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:06.997200 20901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:06.997435 20901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:07.002677 20907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:07.002677 20909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:07.002700 20906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:07.003163 20901 server_base.cc:1034] running on GCE node
I20260927 04:03:07.003422 20901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:07.003643 20901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:07.004797 20901 hybrid_clock.cc:648] HybridClock initialized: now 1790481787004780 us; error 32 us; skew 500 ppm
Sep 27 04:03:07 dist-test-slave-zqrn krb5kdc[20526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481787, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:07.007543 20901 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:03:07.008020 20901 webserver.cc:460] Webserver started at http://127.17.184.195:38579/ using document root <none> and password file <none>
I20260927 04:03:07.008270 20901 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:07.008347 20901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:07.008512 20901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:07.009621 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "6b0a89552c6c416e9ce1e25f0eb14701"
format_stamp: "Formatted at 2026-09-27 04:03:07 on dist-test-slave-zqrn"
server_key: "d4240c19c966e71c5e832a0b179911be"
server_key_iv: "b182abd4bdb67050ed60ec2912fb83d3"
server_key_version: "encryptionkey@0"
I20260927 04:03:07.009997 20901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "6b0a89552c6c416e9ce1e25f0eb14701"
format_stamp: "Formatted at 2026-09-27 04:03:07 on dist-test-slave-zqrn"
server_key: "d4240c19c966e71c5e832a0b179911be"
server_key_iv: "b182abd4bdb67050ed60ec2912fb83d3"
server_key_version: "encryptionkey@0"
I20260927 04:03:07.011415 20901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:07.012213 20916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:07.012435 20901 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:07.012534 20901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "6b0a89552c6c416e9ce1e25f0eb14701"
format_stamp: "Formatted at 2026-09-27 04:03:07 on dist-test-slave-zqrn"
server_key: "d4240c19c966e71c5e832a0b179911be"
server_key_iv: "b182abd4bdb67050ed60ec2912fb83d3"
server_key_version: "encryptionkey@0"
I20260927 04:03:07.012614 20901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:07.029753 20901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:07.032246 20901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:07.032415 20901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:07.032682 20901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:07.033003 20901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:07.033037 20901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:07.033089 20901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:07.033126 20901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:07.039536 20901 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:33241
I20260927 04:03:07.039593 21029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:33241 every 8 connection(s)
I20260927 04:03:07.039925 20901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:07.045871 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 20901
I20260927 04:03:07.046000 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:07.046175 18147 external_mini_cluster.cc:1428] Setting key fe0e2633e34ccd3674a900213db33b94
Sep 27 04:03:07 dist-test-slave-zqrn krb5kdc[20526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481787, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:07.050588 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:07.040698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:46973 (local address 127.17.184.254:43787)
0927 04:03:07.040746 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:07.040748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:07.040945 (+ 197us) server_negotiation.cc:407] Connection header received
0927 04:03:07.041463 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:07.041465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:07.041479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:07.041548 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:07.042500 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:07.043141 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:07.044201 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:07.044412 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:07.044480 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:07.047265 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:07.047267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:07.047267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:07.047275 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:07.048661 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:07.049199 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:07.049200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:07.049201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:07.049276 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:07.049607 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:07.049609 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:07.049609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:07.049833 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:07.049912 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:07.050323 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 04:03:07.050476 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:03:07.051017 21032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:07.040641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:43787 (local address 127.17.184.195:46973)
0927 04:03:07.040806 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:07.040849 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:07.041336 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:07.041603 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:07.041606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:07.042177 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:07.042379 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:07.042383 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:07.043260 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:07.043262 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:07.044095 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:07.044097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:07.044514 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:07.044515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:07.044632 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:07.045293 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:07.045304 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:07.047158 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:07.048747 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:07.048755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:07.048758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:07.049073 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:07.049377 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:07.049377 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:07.049378 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:07.049544 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:07.049945 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:07.049952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:07.050139 (+ 187us) client_negotiation.cc:769] Sending connection context
0927 04:03:07.050287 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 04:03:07.050466 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260927 04:03:07.051893 21030 heartbeater.cc:347] Connected to a master server at 127.17.184.254:43787
I20260927 04:03:07.052006 21030 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:07.052222 21030 heartbeater.cc:511] Master 127.17.184.254:43787 requested a full tablet report, sending...
I20260927 04:03:07.052595 20571 ts_manager.cc:194] Registered new tserver with Master: 6b0a89552c6c416e9ce1e25f0eb14701 (127.17.184.195:33241)
I20260927 04:03:07.052940 20571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:46973
I20260927 04:03:07.058251 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 04:03:07.069666 20611 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:07.068625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.17.184.254:43787)
0927 04:03:07.068705 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:07.068708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:07.068817 (+ 109us) server_negotiation.cc:407] Connection header received
0927 04:03:07.068907 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:07.068908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:07.068918 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:07.068990 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:07.069567 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46210: BlockingRecv error: recv got EOF from 127.0.0.1:46210 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:03:07.070571 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 20633
I20260927 04:03:07.075162 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 20767
I20260927 04:03:07.079628 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 20901
I20260927 04:03:07.084144 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 20539
2026-09-27T04:03:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1891 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:07 dist-test-slave-zqrn krb5kdc[21042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:07 dist-test-slave-zqrn krb5kdc[21042](info): set up 2 sockets
Sep 27 04:03:07 dist-test-slave-zqrn krb5kdc[21042](info): commencing operation
krb5kdc: starting...
W20260927 04:03:08.142987 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:08 dist-test-slave-zqrn krb5kdc[21042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:08.271637 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:36499
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:36499
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:08.356700 21055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:08.356927 21055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:08.356952 21055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:08.358143 21055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:08.358192 21055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:08.358209 21055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:08.358223 21055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:08.358238 21055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:08.359805 21055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:36499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:36499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:08.360071 21055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:08.360338 21055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:08.365589 21060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:08.365603 21061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:08.365589 21063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:08.365706 21055 server_base.cc:1034] running on GCE node
I20260927 04:03:08.366147 21055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:08.366478 21055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:08.367632 21055 hybrid_clock.cc:648] HybridClock initialized: now 1790481788367620 us; error 32 us; skew 500 ppm
Sep 27 04:03:08 dist-test-slave-zqrn krb5kdc[21042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481788, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:08.370432 21055 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:08.371037 21055 webserver.cc:460] Webserver started at http://127.17.184.254:46613/ using document root <none> and password file <none>
I20260927 04:03:08.371280 21055 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:08.371327 21055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:08.371464 21055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:08.372681 21055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "bb98cc294b7144149a9a094ba2e858a9"
format_stamp: "Formatted at 2026-09-27 04:03:08 on dist-test-slave-zqrn"
server_key: "f4f79339665e2709e55c84bb1ee6cf26"
server_key_iv: "797ee1a4b552665b0e6cbe5f625f0cf1"
server_key_version: "encryptionkey@0"
I20260927 04:03:08.373067 21055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "bb98cc294b7144149a9a094ba2e858a9"
format_stamp: "Formatted at 2026-09-27 04:03:08 on dist-test-slave-zqrn"
server_key: "f4f79339665e2709e55c84bb1ee6cf26"
server_key_iv: "797ee1a4b552665b0e6cbe5f625f0cf1"
server_key_version: "encryptionkey@0"
I20260927 04:03:08.374537 21055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:08.375550 21070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:08.375917 21055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:08.376000 21055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "bb98cc294b7144149a9a094ba2e858a9"
format_stamp: "Formatted at 2026-09-27 04:03:08 on dist-test-slave-zqrn"
server_key: "f4f79339665e2709e55c84bb1ee6cf26"
server_key_iv: "797ee1a4b552665b0e6cbe5f625f0cf1"
server_key_version: "encryptionkey@0"
I20260927 04:03:08.376097 21055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:08.387519 21055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:08.389955 21055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:08.390103 21055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:08.394622 21055 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:36499
I20260927 04:03:08.394659 21122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:36499 every 8 connection(s)
I20260927 04:03:08.395244 21055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:08.395964 21123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:08.396726 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 21055
I20260927 04:03:08.396814 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:08.397045 18147 external_mini_cluster.cc:1428] Setting key deddb9134c740d23cf76ae9134cce50c
I20260927 04:03:08.399343 21123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9: Bootstrap starting.
I20260927 04:03:08.399996 21123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:08.400302 21123 log.cc:824] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:08.400998 21123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9: No bootstrap required, opened a new log
I20260927 04:03:08.402817 21123 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb98cc294b7144149a9a094ba2e858a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36499 } }
I20260927 04:03:08.403026 21123 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:08.403056 21123 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb98cc294b7144149a9a094ba2e858a9, State: Initialized, Role: FOLLOWER
I20260927 04:03:08.403208 21123 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb98cc294b7144149a9a094ba2e858a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36499 } }
I20260927 04:03:08.403294 21123 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:08.403317 21123 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:08.403385 21123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:08.404069 21123 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb98cc294b7144149a9a094ba2e858a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36499 } }
I20260927 04:03:08.404187 21123 leader_election.cc:302] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb98cc294b7144149a9a094ba2e858a9; no voters:
Sep 27 04:03:08 dist-test-slave-zqrn krb5kdc[21042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:08.404366 21123 leader_election.cc:288] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:08.404448 21128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:08.404587 21128 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [term 1 LEADER]: Becoming Leader. State: Replica: bb98cc294b7144149a9a094ba2e858a9, State: Running, Role: LEADER
I20260927 04:03:08.404631 21123 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:08.404695 21128 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb98cc294b7144149a9a094ba2e858a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36499 } }
I20260927 04:03:08.405036 21129 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb98cc294b7144149a9a094ba2e858a9. Latest consensus state: current_term: 1 leader_uuid: "bb98cc294b7144149a9a094ba2e858a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb98cc294b7144149a9a094ba2e858a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36499 } } }
I20260927 04:03:08.405126 21129 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:08.405367 21130 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb98cc294b7144149a9a094ba2e858a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb98cc294b7144149a9a094ba2e858a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36499 } } }
I20260927 04:03:08.405496 21130 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:08.405632 21135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:08.406461 21135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:08.409003 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:08.397618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.17.184.254:36499)
0927 04:03:08.397815 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:08.397823 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:08.397851 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:03:08.398299 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:08.398306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:08.398360 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:08.398497 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:08.399484 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:08.400596 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:08.401436 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:08.401866 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:08.401990 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:08.404938 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:08.404943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:08.404944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:08.404957 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:08.407337 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:08.407782 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:08.407785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:08.407785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:08.407887 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:08.408094 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:08.408095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:08.408095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:08.408411 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:08.408562 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:08.408638 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 04:03:08.408797 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260927 04:03:08.410079 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:03:08.416416 21135 catalog_manager.cc:1347] Generated new cluster ID: eb527835a44b473290aa9af61f92ef60
I20260927 04:03:08.416483 21135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:08.429486 21126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:08.425883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35100 (local address 127.17.184.254:36499)
0927 04:03:08.425932 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:08.425935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:08.426071 (+ 136us) server_negotiation.cc:407] Connection header received
0927 04:03:08.426147 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:08.426149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:08.426161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:08.426243 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:08.426627 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:08.427207 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:08.427953 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:08.428167 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:08.428255 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:08.428980 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:08.428982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:08.428982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:08.429187 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:08.429204 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:08.429209 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:08.429273 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:08.429346 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 04:03:08.429370 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:03:08.446385 21135 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:08.447185 21135 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:08.451929 21135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb98cc294b7144149a9a094ba2e858a9: Generated new TSK 0
I20260927 04:03:08.452162 21135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:03:08.461294 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 21055
2026-09-27T04:03:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1375 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:08 dist-test-slave-zqrn krb5kdc[21156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:08 dist-test-slave-zqrn krb5kdc[21156](info): set up 2 sockets
Sep 27 04:03:08 dist-test-slave-zqrn krb5kdc[21156](info): commencing operation
krb5kdc: starting...
W20260927 04:03:09.502852 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:09 dist-test-slave-zqrn krb5kdc[21156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:09.629951 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:37681
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:37681
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:09.713382 21169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:09.713604 21169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:09.713627 21169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:09.714742 21169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:09.714777 21169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:09.714792 21169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:09.714804 21169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:09.714816 21169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:09.716311 21169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:37681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:37681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:09.716511 21169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:09.716687 21169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:09.721828 21174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:09.721836 21175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:09.721875 21169 server_base.cc:1034] running on GCE node
W20260927 04:03:09.722070 21177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:09.722335 21169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:09.722688 21169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:09.723892 21169 hybrid_clock.cc:648] HybridClock initialized: now 1790481789723870 us; error 42 us; skew 500 ppm
Sep 27 04:03:09 dist-test-slave-zqrn krb5kdc[21156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481789, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:09.726876 21169 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:09.727447 21169 webserver.cc:460] Webserver started at http://127.17.184.254:37917/ using document root <none> and password file <none>
I20260927 04:03:09.727707 21169 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:09.727785 21169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:09.727954 21169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:09.729142 21169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "464a797621a642f8ad7cb3b68aaae9f3"
format_stamp: "Formatted at 2026-09-27 04:03:09 on dist-test-slave-zqrn"
server_key: "7a851d493acd7934d8ae04669971389c"
server_key_iv: "1ef89e62d63e9d90fd79abfaa38ac0a0"
server_key_version: "encryptionkey@0"
I20260927 04:03:09.729554 21169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "464a797621a642f8ad7cb3b68aaae9f3"
format_stamp: "Formatted at 2026-09-27 04:03:09 on dist-test-slave-zqrn"
server_key: "7a851d493acd7934d8ae04669971389c"
server_key_iv: "1ef89e62d63e9d90fd79abfaa38ac0a0"
server_key_version: "encryptionkey@0"
I20260927 04:03:09.731112 21169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 04:03:09.732021 21184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:09.732237 21169 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:09.732359 21169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "464a797621a642f8ad7cb3b68aaae9f3"
format_stamp: "Formatted at 2026-09-27 04:03:09 on dist-test-slave-zqrn"
server_key: "7a851d493acd7934d8ae04669971389c"
server_key_iv: "1ef89e62d63e9d90fd79abfaa38ac0a0"
server_key_version: "encryptionkey@0"
I20260927 04:03:09.732447 21169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:09.748123 21169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:09.750193 21169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:09.750360 21169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:09.754598 21169 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:37681
I20260927 04:03:09.754633 21236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:37681 every 8 connection(s)
I20260927 04:03:09.755101 21169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:09.756134 21237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:09.758358 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3: Bootstrap starting.
I20260927 04:03:09.758975 21237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:09.759228 21237 log.cc:824] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:09.759944 21237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3: No bootstrap required, opened a new log
I20260927 04:03:09.761440 21237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a797621a642f8ad7cb3b68aaae9f3" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37681 } }
I20260927 04:03:09.761575 21237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:09.761597 21237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464a797621a642f8ad7cb3b68aaae9f3, State: Initialized, Role: FOLLOWER
I20260927 04:03:09.761747 21237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a797621a642f8ad7cb3b68aaae9f3" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37681 } }
I20260927 04:03:09.761838 21237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:09.761860 21237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:09.761890 21237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:09.762629 21237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a797621a642f8ad7cb3b68aaae9f3" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37681 } }
I20260927 04:03:09.762713 21237 leader_election.cc:302] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 464a797621a642f8ad7cb3b68aaae9f3; no voters:
I20260927 04:03:09.762850 21237 leader_election.cc:288] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:09.762944 21240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:09.763113 21240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [term 1 LEADER]: Becoming Leader. State: Replica: 464a797621a642f8ad7cb3b68aaae9f3, State: Running, Role: LEADER
I20260927 04:03:09.763139 21237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:09.763206 21240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a797621a642f8ad7cb3b68aaae9f3" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37681 } }
I20260927 04:03:09.763634 21241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "464a797621a642f8ad7cb3b68aaae9f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a797621a642f8ad7cb3b68aaae9f3" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37681 } } }
I20260927 04:03:09.763728 21241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:09.763926 21242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464a797621a642f8ad7cb3b68aaae9f3. Latest consensus state: current_term: 1 leader_uuid: "464a797621a642f8ad7cb3b68aaae9f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464a797621a642f8ad7cb3b68aaae9f3" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37681 } } }
I20260927 04:03:09.764011 21242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:09.763960 21249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:09.764607 21249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:09.765106 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 21169
I20260927 04:03:09.765216 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:09.765384 18147 external_mini_cluster.cc:1428] Setting key 50af376310e7531ef2842e4cb35b12b6
I20260927 04:03:09.766456 21249 catalog_manager.cc:1347] Generated new cluster ID: 105f38e3b3d84a74976857a74d6b980e
I20260927 04:03:09.766513 21249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:03:09 dist-test-slave-zqrn krb5kdc[21156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:09.775277 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:09.765863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56316 (local address 127.17.184.254:37681)
0927 04:03:09.766132 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:09.766142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:09.766183 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:03:09.766296 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:09.766303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:09.766359 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:09.766611 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:09.767261 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:09.768205 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:09.768979 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:09.769428 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:09.769549 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:09.771883 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:09.771887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:09.771887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:09.771899 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:09.773631 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:09.774045 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:09.774048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:09.774049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:09.774166 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:09.774379 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:09.774380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:09.774381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:09.774630 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:09.774817 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:09.774884 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 04:03:09.775053 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":73,"threads_started":1}
I20260927 04:03:09.776331 21249 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:09.776810 21249 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:09.782177 21249 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 464a797621a642f8ad7cb3b68aaae9f3: Generated new TSK 0
I20260927 04:03:09.782392 21249 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:03:09.822985 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37681
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:09.925843 21263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:09.926105 21263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:09.926147 21263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:09.927484 21263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:09.927551 21263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:09.927597 21263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:09.929245 21263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:09.929517 21263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:09.929757 21263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:09.935906 21269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:09.936049 21263 server_base.cc:1034] running on GCE node
W20260927 04:03:09.935900 21271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:09.936067 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:09.936420 21263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:09.936707 21263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:09.937863 21263 hybrid_clock.cc:648] HybridClock initialized: now 1790481789937850 us; error 29 us; skew 500 ppm
Sep 27 04:03:09 dist-test-slave-zqrn krb5kdc[21156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481789, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:09.940569 21263 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:03:09.941138 21263 webserver.cc:460] Webserver started at http://127.17.184.193:34899/ using document root <none> and password file <none>
I20260927 04:03:09.941371 21263 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:09.941438 21263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:09.941594 21263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:09.942677 21263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "e940623fd6c54717905749168417bef9"
format_stamp: "Formatted at 2026-09-27 04:03:09 on dist-test-slave-zqrn"
server_key: "eb3402d8f89dcffca08983bf178d3bd7"
server_key_iv: "d3e5c2e36656eae27b6384bd1c706a51"
server_key_version: "encryptionkey@0"
I20260927 04:03:09.943105 21263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "e940623fd6c54717905749168417bef9"
format_stamp: "Formatted at 2026-09-27 04:03:09 on dist-test-slave-zqrn"
server_key: "eb3402d8f89dcffca08983bf178d3bd7"
server_key_iv: "d3e5c2e36656eae27b6384bd1c706a51"
server_key_version: "encryptionkey@0"
I20260927 04:03:09.944456 21263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:09.945341 21278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:09.945513 21263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:09.945617 21263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "e940623fd6c54717905749168417bef9"
format_stamp: "Formatted at 2026-09-27 04:03:09 on dist-test-slave-zqrn"
server_key: "eb3402d8f89dcffca08983bf178d3bd7"
server_key_iv: "d3e5c2e36656eae27b6384bd1c706a51"
server_key_version: "encryptionkey@0"
I20260927 04:03:09.945691 21263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:09.955832 21263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:09.957899 21263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:09.958061 21263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:09.958302 21263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:09.958607 21263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:09.958660 21263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:09.958704 21263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:09.958756 21263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:09.965013 21263 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:41603
I20260927 04:03:09.965076 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:41603 every 8 connection(s)
I20260927 04:03:09.965445 21263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:09.969058 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 21263
I20260927 04:03:09.969146 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:09.969388 18147 external_mini_cluster.cc:1428] Setting key c11e28f2d2b7e5d68aa3a9953da711fd
Sep 27 04:03:09 dist-test-slave-zqrn krb5kdc[21156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481789, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:09.976753 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:09.966258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:39027 (local address 127.17.184.254:37681)
0927 04:03:09.966330 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:09.966333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:09.966560 (+ 227us) server_negotiation.cc:407] Connection header received
0927 04:03:09.967193 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:09.967195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:09.967212 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:09.967270 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:09.968286 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:09.968819 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:09.970117 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:09.970474 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:09.970547 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:09.973130 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:09.973131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:09.973132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:09.973139 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:09.975007 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:09.975591 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:09.975593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:09.975593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:09.975651 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:09.975990 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:09.975991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:09.975992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:09.976139 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:09.976205 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:09.976585 (+ 380us) server_negotiation.cc:299] Negotiation successful
0927 04:03:09.976666 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:03:09.977035 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:09.966255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37681 (local address 127.17.184.193:39027)
0927 04:03:09.966428 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:09.966464 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:09.967081 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:09.967371 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:09.967374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:09.967955 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:09.968155 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:09.968161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:09.968919 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:09.968921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:09.969997 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:09.969999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:09.970565 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:09.970566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:09.970699 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:09.971375 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:09.971384 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:09.973003 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:09.975094 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:09.975104 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:09.975106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:09.975483 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:09.975726 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:09.975727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:09.975728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:09.975892 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:09.976224 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:09.976231 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:09.976422 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 04:03:09.976552 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 04:03:09.976753 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260927 04:03:09.977766 21392 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37681
I20260927 04:03:09.977909 21392 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:09.978122 21392 heartbeater.cc:511] Master 127.17.184.254:37681 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:03:09.978675 21201 ts_manager.cc:194] Registered new tserver with Master: e940623fd6c54717905749168417bef9 (127.17.184.193:41603)
I20260927 04:03:09.979337 21201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:39027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:03:09.998879 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37681
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:10.087570 21397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:10.087801 21397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:10.087852 21397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:10.089047 21397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:10.089113 21397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:10.089157 21397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:10.090636 21397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:10.090969 21397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:10.091205 21397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:10.096552 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:10.096642 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:10.096666 21397 server_base.cc:1034] running on GCE node
W20260927 04:03:10.096565 21405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:10.097059 21397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:10.097347 21397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:10.098541 21397 hybrid_clock.cc:648] HybridClock initialized: now 1790481790098512 us; error 41 us; skew 500 ppm
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481790, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:10.101349 21397 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:03:10.101859 21397 webserver.cc:460] Webserver started at http://127.17.184.194:41775/ using document root <none> and password file <none>
I20260927 04:03:10.102116 21397 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:10.102186 21397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:10.102336 21397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:10.103469 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "ce3419d4d3ab4dcd9289d89660aaaf8e"
format_stamp: "Formatted at 2026-09-27 04:03:10 on dist-test-slave-zqrn"
server_key: "97bfd623eef72c8d2c2167c2880cde15"
server_key_iv: "99f9b121247f47d5fecd3bdd113831e1"
server_key_version: "encryptionkey@0"
I20260927 04:03:10.103881 21397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "ce3419d4d3ab4dcd9289d89660aaaf8e"
format_stamp: "Formatted at 2026-09-27 04:03:10 on dist-test-slave-zqrn"
server_key: "97bfd623eef72c8d2c2167c2880cde15"
server_key_iv: "99f9b121247f47d5fecd3bdd113831e1"
server_key_version: "encryptionkey@0"
I20260927 04:03:10.105209 21397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 04:03:10.106143 21412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:10.106330 21397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:10.106424 21397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "ce3419d4d3ab4dcd9289d89660aaaf8e"
format_stamp: "Formatted at 2026-09-27 04:03:10 on dist-test-slave-zqrn"
server_key: "97bfd623eef72c8d2c2167c2880cde15"
server_key_iv: "99f9b121247f47d5fecd3bdd113831e1"
server_key_version: "encryptionkey@0"
I20260927 04:03:10.106513 21397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:10.121316 21397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:10.124261 21397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:10.124437 21397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:10.124699 21397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:10.125063 21397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:10.125120 21397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:10.125171 21397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:10.125206 21397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:10.131146 21397 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33389
I20260927 04:03:10.131183 21525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33389 every 8 connection(s)
I20260927 04:03:10.131531 21397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:10.134328 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 21397
I20260927 04:03:10.134443 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:10.134629 18147 external_mini_cluster.cc:1428] Setting key bd95fc09c4dd06a7060b4de8a226f43f
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481790, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:10.141801 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:10.132373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:52851 (local address 127.17.184.254:37681)
0927 04:03:10.132451 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:10.132453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:10.132657 (+ 204us) server_negotiation.cc:407] Connection header received
0927 04:03:10.133115 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:10.133117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:10.133128 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:10.133195 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:10.134128 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.134665 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.135952 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.136190 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.136275 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:10.138760 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:10.138762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:10.138763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:10.138772 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:10.140353 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:10.140785 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:10.140786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:10.140787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:10.140854 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:10.141114 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:10.141115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:10.141115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:10.141246 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:10.141326 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:10.141611 (+ 285us) server_negotiation.cc:299] Negotiation successful
0927 04:03:10.141703 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 04:03:10.142017 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:10.132273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37681 (local address 127.17.184.194:52851)
0927 04:03:10.132450 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:10.132493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:10.132994 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:10.133267 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:10.133271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:10.133835 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:10.134036 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:10.134041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.134778 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.134780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:10.135837 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:10.135839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.136286 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.136287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:10.136420 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:10.136977 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:10.136989 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:10.138648 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:10.140456 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:10.140463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:10.140465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:10.140706 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:10.140921 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:10.140922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:10.140922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:10.141042 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:10.141316 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:10.141323 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:10.141477 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 04:03:10.141589 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 04:03:10.141763 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260927 04:03:10.142571 21526 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37681
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
I20260927 04:03:10.142660 21526 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:10.142869 21526 heartbeater.cc:511] Master 127.17.184.254:37681 requested a full tablet report, sending...
I20260927 04:03:10.143342 21201 ts_manager.cc:194] Registered new tserver with Master: ce3419d4d3ab4dcd9289d89660aaaf8e (127.17.184.194:33389)
I20260927 04:03:10.143843 21201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:52851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:03:10.162358 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37681
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:10.244652 21531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:10.244874 21531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:10.244897 21531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:10.245999 21531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:10.246033 21531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:10.246058 21531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:10.247557 21531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:10.247807 21531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:10.248029 21531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:10.253427 21537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:10.253418 21536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:10.253432 21539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:10.253640 21531 server_base.cc:1034] running on GCE node
I20260927 04:03:10.253954 21531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:10.254233 21531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:10.255398 21531 hybrid_clock.cc:648] HybridClock initialized: now 1790481790255379 us; error 37 us; skew 500 ppm
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481790, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:10.258134 21531 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:03:10.258641 21531 webserver.cc:460] Webserver started at http://127.17.184.195:39179/ using document root <none> and password file <none>
I20260927 04:03:10.258879 21531 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:10.259016 21531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:10.259176 21531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:10.260382 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "155a0d8032c447258c7c5e7494326038"
format_stamp: "Formatted at 2026-09-27 04:03:10 on dist-test-slave-zqrn"
server_key: "18122317915d46d43d3aee91a704c90a"
server_key_iv: "268a780ae9be45a317db38c8009168e0"
server_key_version: "encryptionkey@0"
I20260927 04:03:10.260754 21531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "155a0d8032c447258c7c5e7494326038"
format_stamp: "Formatted at 2026-09-27 04:03:10 on dist-test-slave-zqrn"
server_key: "18122317915d46d43d3aee91a704c90a"
server_key_iv: "268a780ae9be45a317db38c8009168e0"
server_key_version: "encryptionkey@0"
I20260927 04:03:10.262158 21531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:10.263041 21546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:10.263209 21531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:10.263314 21531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "155a0d8032c447258c7c5e7494326038"
format_stamp: "Formatted at 2026-09-27 04:03:10 on dist-test-slave-zqrn"
server_key: "18122317915d46d43d3aee91a704c90a"
server_key_iv: "268a780ae9be45a317db38c8009168e0"
server_key_version: "encryptionkey@0"
I20260927 04:03:10.263402 21531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:10.274268 21531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:10.276526 21531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:10.276691 21531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:10.276991 21531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:10.277305 21531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:10.277360 21531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:10.277405 21531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:10.277431 21531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:10.283922 21531 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:38439
I20260927 04:03:10.283960 21659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:38439 every 8 connection(s)
I20260927 04:03:10.284297 21531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:10.287892 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 21531
I20260927 04:03:10.287984 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:10.288149 18147 external_mini_cluster.cc:1428] Setting key 3238093dbb776cfe1710c4bb8d2ee320
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481790, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:10.294239 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:10.285084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:60327 (local address 127.17.184.254:37681)
0927 04:03:10.285132 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:10.285134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:10.285317 (+ 183us) server_negotiation.cc:407] Connection header received
0927 04:03:10.285870 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:10.285872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:10.285882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:10.285948 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:10.286999 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.287495 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.289202 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.289392 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.289447 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:10.291703 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:10.291705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:10.291706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:10.291717 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:10.293000 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:10.293402 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:10.293403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:10.293403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:10.293459 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:10.293669 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:10.293669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:10.293670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:10.293795 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:10.293847 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:10.294106 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 04:03:10.294167 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:03:10.294452 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:10.285006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37681 (local address 127.17.184.195:60327)
0927 04:03:10.285171 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:10.285218 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:10.285766 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:10.286022 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:10.286026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:10.286616 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:10.286859 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:10.286864 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.287739 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.287740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:10.289118 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:10.289121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.289464 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.289465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:10.289567 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:10.290132 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:10.290165 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:10.291613 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:10.293076 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:10.293082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:10.293083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:10.293332 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:10.293518 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:10.293518 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:10.293519 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:10.293617 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:10.293859 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:10.293865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:10.293991 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 04:03:10.294081 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 04:03:10.294234 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260927 04:03:10.295198 21660 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37681
I20260927 04:03:10.295326 21660 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:10.295528 21660 heartbeater.cc:511] Master 127.17.184.254:37681 requested a full tablet report, sending...
I20260927 04:03:10.295980 21201 ts_manager.cc:194] Registered new tserver with Master: 155a0d8032c447258c7c5e7494326038 (127.17.184.195:38439)
I20260927 04:03:10.296337 21201 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:60327
I20260927 04:03:10.300091 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481790, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21156](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481790, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:10.323945 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:10.316363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56322 (local address 127.17.184.254:37681)
0927 04:03:10.316449 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:10.316451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:10.316571 (+ 120us) server_negotiation.cc:407] Connection header received
0927 04:03:10.316701 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:10.316702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:10.316713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:10.316783 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:10.317385 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.317884 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.318654 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:10.318848 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:10.318983 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:10.321308 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:10.321309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:10.321310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:10.321318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:10.322648 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:10.323110 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:10.323112 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:10.323113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:10.323174 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:10.323374 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:10.323375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:10.323376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:10.323532 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:10.323693 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:10.323758 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 04:03:10.323838 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 04:03:10.324239 21201 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56322
I20260927 04:03:10.324734 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 21263
I20260927 04:03:10.329039 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 21397
I20260927 04:03:10.333423 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 21531
I20260927 04:03:10.338079 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 21169
2026-09-27T04:03:10Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1878 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21672](info): set up 2 sockets
Sep 27 04:03:10 dist-test-slave-zqrn krb5kdc[21672](info): commencing operation
krb5kdc: starting...
W20260927 04:03:11.378975 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:11 dist-test-slave-zqrn krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:11.505844 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:42063
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:42063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:11.590889 21685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:11.591120 21685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:11.591142 21685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:11.592247 21685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:11.592281 21685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:11.592296 21685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:11.592308 21685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:11.592341 21685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:11.593801 21685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:42063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:42063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:11.594005 21685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:11.594184 21685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:11.599210 21693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:11.599231 21690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:11.599222 21691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:11.599396 21685 server_base.cc:1034] running on GCE node
I20260927 04:03:11.599777 21685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:11.600095 21685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:11.601254 21685 hybrid_clock.cc:648] HybridClock initialized: now 1790481791601234 us; error 39 us; skew 500 ppm
Sep 27 04:03:11 dist-test-slave-zqrn krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481791, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:11.604089 21685 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:11.604513 21685 webserver.cc:460] Webserver started at http://127.17.184.254:42887/ using document root <none> and password file <none>
I20260927 04:03:11.604738 21685 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:11.604794 21685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:11.604884 21685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:11.605968 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "875820e73f5e4284b6771a860e5e865f"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "29e6b95fd61ff16289191a86e87a2adf"
server_key_iv: "3f3501c2b15177eda6416cc11b9683a9"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.606287 21685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "875820e73f5e4284b6771a860e5e865f"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "29e6b95fd61ff16289191a86e87a2adf"
server_key_iv: "3f3501c2b15177eda6416cc11b9683a9"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.607734 21685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:11.608529 21700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:11.608795 21685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:11.608896 21685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "875820e73f5e4284b6771a860e5e865f"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "29e6b95fd61ff16289191a86e87a2adf"
server_key_iv: "3f3501c2b15177eda6416cc11b9683a9"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.608985 21685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:11.629077 21685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:11.631531 21685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:11.631709 21685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:11.635771 21685 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:42063
I20260927 04:03:11.635810 21752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:42063 every 8 connection(s)
I20260927 04:03:11.636336 21685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:11.637080 21753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:11.639505 21753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f: Bootstrap starting.
I20260927 04:03:11.640131 21753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:11.640435 21753 log.cc:824] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:11.641186 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 21685
I20260927 04:03:11.641304 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:11.641427 21753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f: No bootstrap required, opened a new log
I20260927 04:03:11.641500 18147 external_mini_cluster.cc:1428] Setting key 03cc9375fc35db48a33330acc25000f5
I20260927 04:03:11.644644 21753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875820e73f5e4284b6771a860e5e865f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42063 } }
I20260927 04:03:11.644790 21753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:11.644814 21753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 875820e73f5e4284b6771a860e5e865f, State: Initialized, Role: FOLLOWER
I20260927 04:03:11.644941 21753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875820e73f5e4284b6771a860e5e865f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42063 } }
I20260927 04:03:11.645020 21753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:11.645069 21753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:11.645115 21753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:11.645843 21753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875820e73f5e4284b6771a860e5e865f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42063 } }
I20260927 04:03:11.645953 21753 leader_election.cc:302] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 875820e73f5e4284b6771a860e5e865f; no voters:
I20260927 04:03:11.646095 21753 leader_election.cc:288] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:11.646186 21758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:11.646323 21758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [term 1 LEADER]: Becoming Leader. State: Replica: 875820e73f5e4284b6771a860e5e865f, State: Running, Role: LEADER
I20260927 04:03:11.646338 21753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:11.646412 21758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875820e73f5e4284b6771a860e5e865f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42063 } }
I20260927 04:03:11.646780 21758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "875820e73f5e4284b6771a860e5e865f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875820e73f5e4284b6771a860e5e865f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42063 } } }
I20260927 04:03:11.646857 21758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:11.646988 21758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 875820e73f5e4284b6771a860e5e865f. Latest consensus state: current_term: 1 leader_uuid: "875820e73f5e4284b6771a860e5e865f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "875820e73f5e4284b6771a860e5e865f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42063 } } }
I20260927 04:03:11.647040 21758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:11.647178 21765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:11.647720 21765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:11.649236 21765 catalog_manager.cc:1347] Generated new cluster ID: 1b74ba671ed84c29b3d30ecc6d354303
I20260927 04:03:11.649289 21765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:03:11 dist-test-slave-zqrn krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:11.654254 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:11.642056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.17.184.254:42063)
0927 04:03:11.642244 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:11.642253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:11.642279 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:03:11.643092 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:11.643102 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:11.643159 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:11.643295 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:11.644148 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:11.645276 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:11.646348 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:11.647109 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:11.647231 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:11.650499 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:11.650503 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:11.650503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:11.650516 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:11.652597 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:11.652985 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:11.652988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:11.652989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:11.653096 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:11.653296 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:11.653297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:11.653298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:11.653609 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:11.653758 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:11.653886 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 04:03:11.654031 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260927 04:03:11.659304 21765 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:11.659978 21765 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:11.675473 21765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 875820e73f5e4284b6771a860e5e865f: Generated new TSK 0
I20260927 04:03:11.675709 21765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:03:11.711565 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42063
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:11.796386 21779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:11.796605 21779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:11.796629 21779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:11.797858 21779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:11.797904 21779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:11.797933 21779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:11.799577 21779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:11.799847 21779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:11.800082 21779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:11.805388 21784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:11.805388 21787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:11.805421 21785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:11.805804 21779 server_base.cc:1034] running on GCE node
I20260927 04:03:11.806026 21779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:11.806273 21779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:11.807454 21779 hybrid_clock.cc:648] HybridClock initialized: now 1790481791807448 us; error 35 us; skew 500 ppm
Sep 27 04:03:11 dist-test-slave-zqrn krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481791, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:11.816717 21779 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:03:11.817238 21779 webserver.cc:460] Webserver started at http://127.17.184.193:44553/ using document root <none> and password file <none>
I20260927 04:03:11.817503 21779 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:11.817577 21779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:11.817732 21779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:11.818838 21779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "fdc9487b6fe34c7fb1cf84cefadfc198"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "fb3a3dc88c54758e22a65e086f167283"
server_key_iv: "daa0f187d25b9629fef40c8ac7e0dba3"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.819295 21779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "fdc9487b6fe34c7fb1cf84cefadfc198"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "fb3a3dc88c54758e22a65e086f167283"
server_key_iv: "daa0f187d25b9629fef40c8ac7e0dba3"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.820662 21779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:11.821434 21794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:11.821604 21779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:11.821713 21779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "fdc9487b6fe34c7fb1cf84cefadfc198"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "fb3a3dc88c54758e22a65e086f167283"
server_key_iv: "daa0f187d25b9629fef40c8ac7e0dba3"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.821789 21779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:11.837215 21779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:11.839572 21779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:11.839745 21779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:11.839994 21779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:11.840323 21779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:11.840379 21779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:11.840425 21779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:11.840447 21779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:11.846479 21779 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:37965
I20260927 04:03:11.846522 21907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:37965 every 8 connection(s)
I20260927 04:03:11.846875 21779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:11.847246 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 21779
I20260927 04:03:11.847368 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:11.847609 18147 external_mini_cluster.cc:1428] Setting key d11017e2a67e5fa4088c7422453c58a9
Sep 27 04:03:11 dist-test-slave-zqrn krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481791, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:03:11.857515 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:11.847844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:48241 (local address 127.17.184.254:42063)
0927 04:03:11.847903 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:11.847906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:11.848179 (+ 273us) server_negotiation.cc:407] Connection header received
0927 04:03:11.849238 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:11.849239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:11.849251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:11.849291 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:11.850220 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:11.850890 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:11.851897 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:11.852258 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:11.852356 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:11.854483 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:11.854485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:11.854486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:11.854494 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:11.856177 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:11.856543 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:11.856545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:11.856545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:11.856595 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:11.856829 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:11.856829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:11.856830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:11.857039 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:11.857089 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:11.857371 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 04:03:11.857447 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 04:03:11.857645 21910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:11.847853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42063 (local address 127.17.184.193:48241)
0927 04:03:11.848033 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:11.848086 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:11.849123 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:11.849376 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:11.849385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:11.849810 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:11.850098 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:11.850105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:11.851003 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:11.851004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:11.851813 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:11.851815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:11.852356 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:11.852357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:11.852458 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:11.853023 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:11.853032 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:11.854389 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:11.856254 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:11.856260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:11.856262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:11.856473 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:11.856667 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:11.856668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:11.856668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:11.856769 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:11.857092 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:11.857096 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:11.857225 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 04:03:11.857309 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 04:03:11.857427 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260927 04:03:11.858212 21908 heartbeater.cc:347] Connected to a master server at 127.17.184.254:42063
I20260927 04:03:11.858325 21908 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
I20260927 04:03:11.858544 21908 heartbeater.cc:511] Master 127.17.184.254:42063 requested a full tablet report, sending...
I20260927 04:03:11.859200 21717 ts_manager.cc:194] Registered new tserver with Master: fdc9487b6fe34c7fb1cf84cefadfc198 (127.17.184.193:37965)
I20260927 04:03:11.860085 21717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:48241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:03:11.873718 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42063
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:11.955276 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:11.955487 21913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:11.955511 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:11.956611 21913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:11.956645 21913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:11.956671 21913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:11.958200 21913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:11.958400 21913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:11.958576 21913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:03:11.963727 21913 server_base.cc:1034] running on GCE node
W20260927 04:03:11.963634 21921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:11.963665 21919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:11.963634 21918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:11.964082 21913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:11.964344 21913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:11.965513 21913 hybrid_clock.cc:648] HybridClock initialized: now 1790481791965510 us; error 36 us; skew 500 ppm
Sep 27 04:03:11 dist-test-slave-zqrn krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481791, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:11.968045 21913 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:03:11.968575 21913 webserver.cc:460] Webserver started at http://127.17.184.194:34261/ using document root <none> and password file <none>
I20260927 04:03:11.968834 21913 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:11.968904 21913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:11.969041 21913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:11.970476 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "5bd375459b984010936e9a4267ec4167"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "c80d3eccabd3d3cde0fa7b3f9a62553a"
server_key_iv: "1f47e3d8194c452a20267dfcd58d16f3"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.971083 21913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "5bd375459b984010936e9a4267ec4167"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "c80d3eccabd3d3cde0fa7b3f9a62553a"
server_key_iv: "1f47e3d8194c452a20267dfcd58d16f3"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.972864 21913 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:11.973706 21928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:11.974059 21913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:11.974148 21913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "5bd375459b984010936e9a4267ec4167"
format_stamp: "Formatted at 2026-09-27 04:03:11 on dist-test-slave-zqrn"
server_key: "c80d3eccabd3d3cde0fa7b3f9a62553a"
server_key_iv: "1f47e3d8194c452a20267dfcd58d16f3"
server_key_version: "encryptionkey@0"
I20260927 04:03:11.974246 21913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:11.994619 21913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:11.997473 21913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:11.997639 21913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:11.997905 21913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:11.998306 21913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:11.998366 21913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:11.998405 21913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:11.998430 21913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:12.005934 21913 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:39655
I20260927 04:03:12.005991 22041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:39655 every 8 connection(s)
I20260927 04:03:12.006510 21913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:12.009069 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 21913
I20260927 04:03:12.009271 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:12.009549 18147 external_mini_cluster.cc:1428] Setting key e22714e681f9f9e7cad05115b0487f10
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481791, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
I20260927 04:03:12.021389 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:12.007486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:55253 (local address 127.17.184.254:42063)
0927 04:03:12.007552 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:12.007554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:12.007823 (+ 269us) server_negotiation.cc:407] Connection header received
0927 04:03:12.009721 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:12.009723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:12.009739 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:12.009850 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:12.010927 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.011852 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.013437 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.013649 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.013731 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:12.017551 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:12.017553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:12.017554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:12.017564 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:12.019618 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:12.020185 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:12.020187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:12.020187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:12.020255 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:12.020609 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:12.020610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:12.020610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:12.020745 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:12.020840 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:12.021197 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 04:03:12.021290 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:03:12.021682 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:12.007456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42063 (local address 127.17.184.194:55253)
0927 04:03:12.007644 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:12.007704 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:12.009492 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:12.010005 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:12.010011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:12.010432 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:12.010745 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:12.010753 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.011977 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.011979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:12.013322 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:12.013326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.013751 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.013752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:12.013896 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:12.015364 (+ 1468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:12.015374 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:12.017402 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:12.019737 (+ 2335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:12.019747 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:12.019750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:12.020075 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:12.020346 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:12.020347 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:12.020348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:12.020504 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:12.020833 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:12.020840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:12.020994 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 04:03:12.021291 (+ 297us) client_negotiation.cc:240] Negotiation successful
0927 04:03:12.021489 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260927 04:03:12.022255 22042 heartbeater.cc:347] Connected to a master server at 127.17.184.254:42063
I20260927 04:03:12.022393 22042 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:12.022591 22042 heartbeater.cc:511] Master 127.17.184.254:42063 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
I20260927 04:03:12.023365 21717 ts_manager.cc:194] Registered new tserver with Master: 5bd375459b984010936e9a4267ec4167 (127.17.184.194:39655)
I20260927 04:03:12.023808 21717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:55253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:03:12.044267 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42063
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:12.133081 22047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:12.133333 22047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:12.133360 22047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:12.134769 22047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:12.134824 22047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:12.134855 22047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:12.136674 22047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:12.136993 22047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:12.137280 22047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:12.143435 22055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:12.143438 22053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:12.143471 22047 server_base.cc:1034] running on GCE node
W20260927 04:03:12.143438 22052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:12.143971 22047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:12.144233 22047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:12.145383 22047 hybrid_clock.cc:648] HybridClock initialized: now 1790481792145372 us; error 28 us; skew 500 ppm
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481792, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:12.148620 22047 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:03:12.149322 22047 webserver.cc:460] Webserver started at http://127.17.184.195:35911/ using document root <none> and password file <none>
I20260927 04:03:12.149610 22047 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:12.149686 22047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:12.149847 22047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:12.151152 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "1e3f3d4d17454d75803edf46feb0beef"
format_stamp: "Formatted at 2026-09-27 04:03:12 on dist-test-slave-zqrn"
server_key: "5ed82fccbe9972c8684700eaca8ddf12"
server_key_iv: "1eadce435e43c43e111adbfe61d30a13"
server_key_version: "encryptionkey@0"
I20260927 04:03:12.151540 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "1e3f3d4d17454d75803edf46feb0beef"
format_stamp: "Formatted at 2026-09-27 04:03:12 on dist-test-slave-zqrn"
server_key: "5ed82fccbe9972c8684700eaca8ddf12"
server_key_iv: "1eadce435e43c43e111adbfe61d30a13"
server_key_version: "encryptionkey@0"
I20260927 04:03:12.152935 22047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:12.153759 22062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:12.154007 22047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:12.154140 22047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "1e3f3d4d17454d75803edf46feb0beef"
format_stamp: "Formatted at 2026-09-27 04:03:12 on dist-test-slave-zqrn"
server_key: "5ed82fccbe9972c8684700eaca8ddf12"
server_key_iv: "1eadce435e43c43e111adbfe61d30a13"
server_key_version: "encryptionkey@0"
I20260927 04:03:12.154230 22047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:12.166128 22047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:12.168810 22047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:12.168929 22047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:12.169185 22047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:12.169538 22047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:12.169574 22047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:12.169596 22047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:12.169610 22047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:12.176035 22047 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:37873
I20260927 04:03:12.176082 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:37873 every 8 connection(s)
I20260927 04:03:12.176451 22047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:12.180224 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 22047
I20260927 04:03:12.180400 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:12.180641 18147 external_mini_cluster.cc:1428] Setting key 74f205e694b358e2426d2ac0e0a7f538
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481792, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:12.187320 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:12.177426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:38025 (local address 127.17.184.254:42063)
0927 04:03:12.177511 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:12.177513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:12.177741 (+ 228us) server_negotiation.cc:407] Connection header received
0927 04:03:12.178979 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:12.178981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:12.178994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:12.179066 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:12.179825 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.180353 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.181822 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.182072 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.182151 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:12.184422 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:12.184423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:12.184423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:12.184431 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:12.185878 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:12.186282 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:12.186284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:12.186284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:12.186342 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:12.186646 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:12.186647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:12.186647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:12.186770 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:12.186844 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:12.187179 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 04:03:12.187248 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 04:03:12.187572 22178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:12.177391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42063 (local address 127.17.184.195:38025)
0927 04:03:12.177583 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:12.177632 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:12.178781 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:12.179157 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:12.179161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:12.179481 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:12.179715 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:12.179720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.180561 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.180562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:12.181718 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:12.181721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.182170 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.182171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:12.182304 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:12.182812 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:12.182820 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:12.184327 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:12.185961 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:12.185968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:12.185970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:12.186191 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:12.186441 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:12.186442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:12.186442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:12.186556 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:12.186851 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:12.186855 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:12.187054 (+ 199us) client_negotiation.cc:769] Sending connection context
0927 04:03:12.187147 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 04:03:12.187292 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260927 04:03:12.188217 22176 heartbeater.cc:347] Connected to a master server at 127.17.184.254:42063
I20260927 04:03:12.188316 22176 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:12.188467 22176 heartbeater.cc:511] Master 127.17.184.254:42063 requested a full tablet report, sending...
I20260927 04:03:12.188898 21717 ts_manager.cc:194] Registered new tserver with Master: 1e3f3d4d17454d75803edf46feb0beef (127.17.184.195:37873)
I20260927 04:03:12.189270 21717 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:38025
I20260927 04:03:12.192902 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:03:12.219681 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:12.209796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.17.184.193:37965)
0927 04:03:12.209988 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:12.209994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:12.210005 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:12.210032 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:12.210034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:12.210049 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:12.210119 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:12.210793 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.211633 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.212545 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.212959 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.213019 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:12.215573 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:12.215581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:12.215582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:12.215589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:12.217315 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:12.217728 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:12.217732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:12.217732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:12.217818 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:12.218097 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:12.218098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:12.218098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:12.218296 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:12.218442 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:12.218620 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 04:03:12.218702 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":102,"threads_started":1}
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:12.231019 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:12.222307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.17.184.254:42063)
0927 04:03:12.222400 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:12.222402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:12.222414 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:03:12.222486 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:12.222487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:12.222496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:12.222559 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:12.223257 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.223812 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.224617 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:12.224903 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:12.224969 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:12.227472 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:12.227474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:12.227474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:12.227482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:12.229153 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:12.229558 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:12.229559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:12.229559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:12.229677 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:12.230210 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:12.230211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:12.230212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:12.230434 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:12.230536 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:12.230768 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 04:03:12.230847 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 04:03:12.231344 21717 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32942
I20260927 04:03:12.231813 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 21779
I20260927 04:03:12.237221 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 21913
I20260927 04:03:12.241743 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 22047
I20260927 04:03:12.246290 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 21685
2026-09-27T04:03:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1907 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[22188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[22188](info): set up 2 sockets
Sep 27 04:03:12 dist-test-slave-zqrn krb5kdc[22188](info): commencing operation
krb5kdc: starting...
W20260927 04:03:13.295363 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:13 dist-test-slave-zqrn krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:13.425508 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44921
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:13.506992 22201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:13.507195 22201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:13.507218 22201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:13.508334 22201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:13.508371 22201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:13.508385 22201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:13.508399 22201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:13.508410 22201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:13.509836 22201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:13.510078 22201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:13.510308 22201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:13.515193 22207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:13.515262 22206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:13.515189 22209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:13.515502 22201 server_base.cc:1034] running on GCE node
I20260927 04:03:13.515772 22201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:13.516016 22201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:13.517158 22201 hybrid_clock.cc:648] HybridClock initialized: now 1790481793517144 us; error 30 us; skew 500 ppm
Sep 27 04:03:13 dist-test-slave-zqrn krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481793, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:13.524693 22201 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:13.525125 22201 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:13.525189 22201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:13.525301 22201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:13.526428 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "fdf56315e66c408292b19c69adb19e94"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "4f1b906495f2ea30c8f511908adfd82c"
server_key_iv: "1fb12321708cfe50e2c08ec075e85393"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.526750 22201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "fdf56315e66c408292b19c69adb19e94"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "4f1b906495f2ea30c8f511908adfd82c"
server_key_iv: "1fb12321708cfe50e2c08ec075e85393"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.528198 22201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:13.529002 22215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:13.529249 22201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:13.529372 22201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "fdf56315e66c408292b19c69adb19e94"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "4f1b906495f2ea30c8f511908adfd82c"
server_key_iv: "1fb12321708cfe50e2c08ec075e85393"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.529470 22201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:13.537789 22201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:13.540427 22201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:13.540612 22201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:13.545317 22201 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44921
I20260927 04:03:13.545342 22267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44921 every 8 connection(s)
I20260927 04:03:13.545667 22201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:13.546628 22268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:13.549045 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94: Bootstrap starting.
I20260927 04:03:13.549651 22268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:13.549957 22268 log.cc:824] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:13.550762 22268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94: No bootstrap required, opened a new log
I20260927 04:03:13.550884 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 22201
I20260927 04:03:13.551061 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:13.551280 18147 external_mini_cluster.cc:1428] Setting key 6531ba4ebfd8c01ae2df3bbaa0f5f206
I20260927 04:03:13.552793 22268 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf56315e66c408292b19c69adb19e94" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44921 } }
I20260927 04:03:13.552955 22268 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:13.553000 22268 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf56315e66c408292b19c69adb19e94, State: Initialized, Role: FOLLOWER
I20260927 04:03:13.553126 22268 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf56315e66c408292b19c69adb19e94" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44921 } }
I20260927 04:03:13.553236 22268 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:13.553274 22268 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:13.553316 22268 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:13.553839 22268 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf56315e66c408292b19c69adb19e94" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44921 } }
I20260927 04:03:13.553958 22268 leader_election.cc:302] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdf56315e66c408292b19c69adb19e94; no voters:
I20260927 04:03:13.554133 22268 leader_election.cc:288] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:13.554194 22273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:13.554332 22273 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [term 1 LEADER]: Becoming Leader. State: Replica: fdf56315e66c408292b19c69adb19e94, State: Running, Role: LEADER
I20260927 04:03:13.554409 22268 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:13.554422 22273 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf56315e66c408292b19c69adb19e94" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44921 } }
I20260927 04:03:13.554759 22273 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdf56315e66c408292b19c69adb19e94. Latest consensus state: current_term: 1 leader_uuid: "fdf56315e66c408292b19c69adb19e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf56315e66c408292b19c69adb19e94" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44921 } } }
I20260927 04:03:13.555003 22273 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:13.555476 22279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:13.554770 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdf56315e66c408292b19c69adb19e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdf56315e66c408292b19c69adb19e94" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44921 } } }
I20260927 04:03:13.555763 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:13.555956 22279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:13.557822 22279 catalog_manager.cc:1347] Generated new cluster ID: 021a54f8a3fc4f1586490f290d809286
I20260927 04:03:13.557860 22279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:03:13 dist-test-slave-zqrn krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:13.563616 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:13.551855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.17.184.254:44921)
0927 04:03:13.552075 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:13.552083 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:13.552109 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:03:13.552804 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:13.552813 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:13.552866 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:13.553039 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:13.553857 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.555305 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.556338 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.556927 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.557042 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:13.559787 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:13.559792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:13.559792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:13.559805 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:13.561823 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:13.562223 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:13.562227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:13.562227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:13.562351 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:13.562556 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:13.562557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:13.562557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:13.562942 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:13.563111 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:13.563194 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 04:03:13.563348 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260927 04:03:13.568189 22279 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:13.568801 22279 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:13.573457 22279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94: Generated new TSK 0
I20260927 04:03:13.573637 22279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:03:13.606595 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44921
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:13.694320 22294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:13.694526 22294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:13.694551 22294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:13.695756 22294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:13.695799 22294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:13.695827 22294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:13.697324 22294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:13.697535 22294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:13.697717 22294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:13.702857 22302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:13.702973 22299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:13.703064 22294 server_base.cc:1034] running on GCE node
W20260927 04:03:13.702857 22300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:13.703425 22294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:13.703639 22294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:13.704857 22294 hybrid_clock.cc:648] HybridClock initialized: now 1790481793704845 us; error 37 us; skew 500 ppm
Sep 27 04:03:13 dist-test-slave-zqrn krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481793, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:13.707628 22294 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:03:13.707988 22294 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:13.708036 22294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:13.708128 22294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:13.709136 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "268c7a0f7e334b2f8074b01bc0bc2652"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "7de689b9f3a4b8b820a1f48911e5fe83"
server_key_iv: "eef09ed9923a03c4e09d60d61fa28060"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.709425 22294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "268c7a0f7e334b2f8074b01bc0bc2652"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "7de689b9f3a4b8b820a1f48911e5fe83"
server_key_iv: "eef09ed9923a03c4e09d60d61fa28060"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.710672 22294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:13.711546 22308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:13.711751 22294 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:13.711823 22294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "268c7a0f7e334b2f8074b01bc0bc2652"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "7de689b9f3a4b8b820a1f48911e5fe83"
server_key_iv: "eef09ed9923a03c4e09d60d61fa28060"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.711915 22294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:13.727201 22294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:13.729609 22294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:13.729763 22294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:13.730021 22294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:13.730360 22294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:13.730399 22294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:13.730422 22294 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:13.730468 22294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:13.737087 22294 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:45527
I20260927 04:03:13.737120 22421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:45527 every 8 connection(s)
I20260927 04:03:13.737426 22294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:13.741786 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 22294
I20260927 04:03:13.741894 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:13.742069 18147 external_mini_cluster.cc:1428] Setting key 57cca393d98e92920a8bdea33bcfd4a9
Sep 27 04:03:13 dist-test-slave-zqrn krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481793, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:13.747805 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:13.738311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:53733 (local address 127.17.184.254:44921)
0927 04:03:13.738372 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:13.738374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:13.738536 (+ 162us) server_negotiation.cc:407] Connection header received
0927 04:03:13.739059 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:13.739060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:13.739074 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:13.739141 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:13.740109 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.740678 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.741771 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.742119 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.742207 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:13.744789 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:13.744791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:13.744792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:13.744801 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:13.746281 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:13.746766 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:13.746767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:13.746768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:13.746830 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:13.747117 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:13.747118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:13.747118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:13.747246 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:13.747318 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:13.747607 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 04:03:13.747701 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 04:03:13.748003 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:13.738236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44921 (local address 127.17.184.193:53733)
0927 04:03:13.738419 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:13.738464 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:13.738976 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:13.739211 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:13.739215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:13.739486 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:13.739991 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:13.739995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.740793 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.740794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:13.741620 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:13.741622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.742296 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.742298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:13.742465 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:13.743163 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:13.743171 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:13.744678 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:13.746394 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:13.746402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:13.746404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:13.746634 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:13.746924 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:13.746925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:13.746926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:13.747043 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:13.747320 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:13.747327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:13.747468 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 04:03:13.747571 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 04:03:13.747745 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":101,"threads_started":1}
I20260927 04:03:13.748550 22422 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44921
I20260927 04:03:13.748649 22422 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:13.748829 22422 heartbeater.cc:511] Master 127.17.184.254:44921 requested a full tablet report, sending...
I20260927 04:03:13.749375 22232 ts_manager.cc:194] Registered new tserver with Master: 268c7a0f7e334b2f8074b01bc0bc2652 (127.17.184.193:45527)
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:03:13.750010 22232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:53733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:03:13.770428 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44921
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:13.853996 22427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:13.854206 22427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:13.854230 22427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:13.855527 22427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:13.855568 22427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:13.855595 22427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:13.857124 22427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:13.857326 22427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:13.857501 22427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:13.862442 22433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:13.862453 22435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:13.862439 22432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:13.862812 22427 server_base.cc:1034] running on GCE node
I20260927 04:03:13.863076 22427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:13.863327 22427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:13.864475 22427 hybrid_clock.cc:648] HybridClock initialized: now 1790481793864462 us; error 27 us; skew 500 ppm
Sep 27 04:03:13 dist-test-slave-zqrn krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481793, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:13.867516 22427 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:03:13.867816 22427 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:13.867858 22427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:13.867949 22427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:13.869001 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "bf3384bf74f14e2e8a8649b7e98e7278"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "19777139224324483dbb7ced71b38b87"
server_key_iv: "fcff1334705ae4e589964737ea940ebd"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.869299 22427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "bf3384bf74f14e2e8a8649b7e98e7278"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "19777139224324483dbb7ced71b38b87"
server_key_iv: "fcff1334705ae4e589964737ea940ebd"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.870532 22427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:13.871335 22441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:13.871531 22427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:13.871604 22427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "bf3384bf74f14e2e8a8649b7e98e7278"
format_stamp: "Formatted at 2026-09-27 04:03:13 on dist-test-slave-zqrn"
server_key: "19777139224324483dbb7ced71b38b87"
server_key_iv: "fcff1334705ae4e589964737ea940ebd"
server_key_version: "encryptionkey@0"
I20260927 04:03:13.871659 22427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:13.887351 22427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:13.889636 22427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:13.889758 22427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:13.889945 22427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:13.890295 22427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:13.890329 22427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:13.890352 22427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:13.890400 22427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:13.896227 22427 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:45943
I20260927 04:03:13.896258 22554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:45943 every 8 connection(s)
I20260927 04:03:13.896535 22427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:03:13 dist-test-slave-zqrn krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481793, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:13.905928 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 22427
I20260927 04:03:13.906055 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:13.906270 18147 external_mini_cluster.cc:1428] Setting key 335d5b1308690e62179156c75b99a1ad
I20260927 04:03:13.907045 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:13.897323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:37615 (local address 127.17.184.254:44921)
0927 04:03:13.897391 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:13.897394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:13.897530 (+ 136us) server_negotiation.cc:407] Connection header received
0927 04:03:13.898011 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:13.898012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:13.898022 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:13.898085 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:13.899221 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.899732 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.900805 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.901002 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.901073 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:13.903669 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:13.903670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:13.903671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:13.903685 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:13.905144 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:13.905561 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:13.905561 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:13.905562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:13.905659 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:13.906178 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:13.906179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:13.906180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:13.906342 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:13.906425 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:13.906806 (+ 381us) server_negotiation.cc:299] Negotiation successful
0927 04:03:13.906931 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 04:03:13.907220 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:13.897224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44921 (local address 127.17.184.194:37615)
0927 04:03:13.897392 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:13.897435 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:13.897930 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:13.898182 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:13.898187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:13.898450 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:13.899050 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:13.899058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.899870 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.899871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:13.900697 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:13.900699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:13.901095 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:13.901096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:13.901216 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:13.901794 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:13.901808 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:13.903548 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:13.905240 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:13.905246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:13.905248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:13.905488 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:13.905928 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:13.905929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:13.905929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:13.906068 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:13.906457 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:13.906463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:13.906653 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 04:03:13.906755 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 04:03:13.906957 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260927 04:03:13.908643 22555 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44921
I20260927 04:03:13.908766 22555 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:13.908937 22555 heartbeater.cc:511] Master 127.17.184.254:44921 requested a full tablet report, sending...
I20260927 04:03:13.909299 22232 ts_manager.cc:194] Registered new tserver with Master: bf3384bf74f14e2e8a8649b7e98e7278 (127.17.184.194:45943)
I20260927 04:03:13.909880 22232 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:37615
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:03:13.934544 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44921
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:14.017858 22560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:14.018117 22560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:14.018158 22560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:14.019438 22560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:14.019512 22560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:14.019559 22560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:14.021139 22560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:14.021430 22560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:14.021678 22560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:03:14.027144 22560 server_base.cc:1034] running on GCE node
W20260927 04:03:14.027122 22566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:14.027115 22568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:14.027110 22565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:14.027557 22560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:14.027776 22560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:14.028963 22560 hybrid_clock.cc:648] HybridClock initialized: now 1790481794028939 us; error 44 us; skew 500 ppm
Sep 27 04:03:14 dist-test-slave-zqrn krb5kdc[22188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:14.031713 22560 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:03:14.032111 22560 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:14.032163 22560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:14.032261 22560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:14.033449 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "692b61b94f9839c9622a12d98faed773"
server_key_iv: "9dde1c302f0a1893f2d5f98a7fff6930"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.033847 22560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "692b61b94f9839c9622a12d98faed773"
server_key_iv: "9dde1c302f0a1893f2d5f98a7fff6930"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.035300 22560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:14.036197 22574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.036477 22560 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:14.036562 22560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "692b61b94f9839c9622a12d98faed773"
server_key_iv: "9dde1c302f0a1893f2d5f98a7fff6930"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.036619 22560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:14.049610 22560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:14.052232 22560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:14.052361 22560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:14.052596 22560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:14.052910 22560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:14.052942 22560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.052965 22560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:14.052978 22560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.059470 22560 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:44051
I20260927 04:03:14.059705 22687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:44051 every 8 connection(s)
I20260927 04:03:14.059877 22560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:03:14 dist-test-slave-zqrn krb5kdc[22188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481794, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:14.069893 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.060680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:37789 (local address 127.17.184.254:44921)
0927 04:03:14.060793 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.060795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.061002 (+ 207us) server_negotiation.cc:407] Connection header received
0927 04:03:14.061449 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.061450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.061461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.061530 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.062517 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.063161 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.064216 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.064417 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.064479 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.066978 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.066979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.066980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:14.066987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.068587 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:14.069021 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:14.069022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:14.069022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:14.069073 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:14.069289 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:14.069289 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:14.069289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:14.069397 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.069447 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.069704 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.069775 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 04:03:14.069881 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 22560
I20260927 04:03:14.069977 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:14.070160 18147 external_mini_cluster.cc:1428] Setting key 43014b9365b213e3480038f3a584fd59
I20260927 04:03:14.070174 22690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.060669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44921 (local address 127.17.184.195:37789)
0927 04:03:14.060842 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.060886 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.061358 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.061585 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.061588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.061918 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:14.062417 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.062422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.063264 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.063265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.064127 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.064129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.064511 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.064512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.064622 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.065168 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:14.065176 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:14.066710 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:14.068666 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:14.068672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:14.068674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:14.068952 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:14.069131 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:14.069132 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:14.069132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:14.069235 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:14.069452 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:14.069456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:14.069580 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.069663 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.069912 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260927 04:03:14.070674 22688 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44921
I20260927 04:03:14.070812 22688 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:14.071033 22688 heartbeater.cc:511] Master 127.17.184.254:44921 requested a full tablet report, sending...
I20260927 04:03:14.071415 22231 ts_manager.cc:194] Registered new tserver with Master: d23ec306d46e48cea3ebf40d8ecfcc3f (127.17.184.195:44051)
I20260927 04:03:14.071813 22231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:37789
I20260927 04:03:14.082131 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:03:14.089814 22272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.083491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.17.184.254:44921)
0927 04:03:14.083553 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.083555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.083670 (+ 115us) server_negotiation.cc:407] Connection header received
0927 04:03:14.083769 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.083770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.083780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.083831 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.084413 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.084969 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.085752 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.085943 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.086000 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.087284 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.087286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.087286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:14.087293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.088716 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:14.089043 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:14.089044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:14.089044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:14.089105 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:14.089310 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:14.089311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:14.089312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:14.089468 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.089548 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.089634 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.089744 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:03:14.091634 22231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:14.093163 22231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:14.100456 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.095903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44051 (local address 127.0.0.1:57354)
0927 04:03:14.096023 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.096034 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.096095 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.096588 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.096589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.096590 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:14.096618 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.096619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.097943 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.097944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.099121 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.099124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.100077 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.100079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.100164 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.100180 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.100319 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.100367 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":92,"threads_started":1}
I20260927 04:03:14.100469 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.096133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.17.184.195:44051)
0927 04:03:14.096334 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.096339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.096349 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:14.096450 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.096453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.096470 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.096590 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:14.096674 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.097775 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.099296 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.099944 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.100077 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.100139 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.100339 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.100382 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260927 04:03:14.100548 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.095750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.17.184.193:45527)
0927 04:03:14.095976 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.095981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.096176 (+ 195us) server_negotiation.cc:407] Connection header received
0927 04:03:14.096220 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.096223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.096239 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.096312 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:14.096594 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.097342 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.099233 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.099973 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.100050 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.100111 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.100416 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.100463 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":64,"threads_started":1}
I20260927 04:03:14.100916 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.095653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:45527 (local address 127.0.0.1:56816)
0927 04:03:14.096012 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.096021 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.096088 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.096407 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.096409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.096412 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:14.096492 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.096494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.097447 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.097448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.099121 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.099124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.100143 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.100144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.100211 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.100215 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.100264 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.100326 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":25,"thread_start_us":139,"threads_started":1}
I20260927 04:03:14.101447 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.095650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:45943 (local address 127.0.0.1:46392)
0927 04:03:14.095939 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.095984 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.096088 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.096408 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.096411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.096412 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:14.096492 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.096495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.098980 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.098981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.100058 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.100061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.101223 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.101224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.101272 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.101274 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.101344 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.101384 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":9,"thread_start_us":92,"threads_started":1}
I20260927 04:03:14.101629 22620 tablet_service.cc:1467] Processing CreateTablet for tablet 5b17325d760a4115b8fd82f544e66711 (DEFAULT_TABLE table=test-table [id=5548d4d067d1465ebcea77de5fcf72e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:14.101758 22356 tablet_service.cc:1467] Processing CreateTablet for tablet 5b17325d760a4115b8fd82f544e66711 (DEFAULT_TABLE table=test-table [id=5548d4d067d1465ebcea77de5fcf72e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:14.102028 22620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b17325d760a4115b8fd82f544e66711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:14.102135 22356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b17325d760a4115b8fd82f544e66711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:14.103137 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.095761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.17.184.194:45943)
0927 04:03:14.095946 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.095951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.096051 (+ 100us) server_negotiation.cc:407] Connection header received
0927 04:03:14.096221 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.096223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.096246 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.096299 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:14.096575 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.097772 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.100159 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.101063 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.101110 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.101166 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.103000 (+ 1834us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.103050 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260927 04:03:14.103852 22489 tablet_service.cc:1467] Processing CreateTablet for tablet 5b17325d760a4115b8fd82f544e66711 (DEFAULT_TABLE table=test-table [id=5548d4d067d1465ebcea77de5fcf72e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:14.104202 22489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b17325d760a4115b8fd82f544e66711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:14.105157 22705 tablet_bootstrap.cc:492] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652: Bootstrap starting.
I20260927 04:03:14.105960 22705 tablet_bootstrap.cc:654] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:14.106710 22705 log.cc:824] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:14.107581 22708 tablet_bootstrap.cc:492] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Bootstrap starting.
I20260927 04:03:14.107674 22705 tablet_bootstrap.cc:492] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652: No bootstrap required, opened a new log
I20260927 04:03:14.107676 22706 tablet_bootstrap.cc:492] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Bootstrap starting.
I20260927 04:03:14.107741 22705 ts_tablet_manager.cc:1397] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:03:14.108237 22706 tablet_bootstrap.cc:654] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:14.108300 22708 tablet_bootstrap.cc:654] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:14.108762 22706 log.cc:824] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:14.108877 22708 log.cc:824] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:14.109745 22708 tablet_bootstrap.cc:492] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: No bootstrap required, opened a new log
I20260927 04:03:14.109830 22708 ts_tablet_manager.cc:1397] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:14.109956 22705 raft_consensus.cc:348] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.110139 22705 raft_consensus.cc:374] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:14.110175 22705 raft_consensus.cc:729] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268c7a0f7e334b2f8074b01bc0bc2652, State: Initialized, Role: FOLLOWER
I20260927 04:03:14.110313 22705 consensus_queue.cc:260] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.110590 22705 ts_tablet_manager.cc:1428] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:03:14.110611 22422 heartbeater.cc:503] Master 127.17.184.254:44921 was elected leader, sending a full tablet report...
I20260927 04:03:14.111368 22706 tablet_bootstrap.cc:492] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: No bootstrap required, opened a new log
I20260927 04:03:14.111438 22706 ts_tablet_manager.cc:1397] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 04:03:14.112011 22708 raft_consensus.cc:348] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.112128 22708 raft_consensus.cc:374] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:14.112147 22708 raft_consensus.cc:729] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3384bf74f14e2e8a8649b7e98e7278, State: Initialized, Role: FOLLOWER
I20260927 04:03:14.112228 22708 consensus_queue.cc:260] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.112526 22555 heartbeater.cc:503] Master 127.17.184.254:44921 was elected leader, sending a full tablet report...
I20260927 04:03:14.112512 22708 ts_tablet_manager.cc:1428] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:03:14.113622 22706 raft_consensus.cc:348] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.113736 22706 raft_consensus.cc:374] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:14.113757 22706 raft_consensus.cc:729] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23ec306d46e48cea3ebf40d8ecfcc3f, State: Initialized, Role: FOLLOWER
I20260927 04:03:14.113837 22706 consensus_queue.cc:260] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.114063 22706 ts_tablet_manager.cc:1428] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:03:14.114097 22688 heartbeater.cc:503] Master 127.17.184.254:44921 was elected leader, sending a full tablet report...
W20260927 04:03:14.147148 22556 tablet.cc:2369] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:03:14.238212 22423 tablet.cc:2369] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:03:14.310509 22689 tablet.cc:2369] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:14.325172 22712 raft_consensus.cc:482] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:14.325297 22712 raft_consensus.cc:504] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.325697 22712 leader_election.cc:288] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 268c7a0f7e334b2f8074b01bc0bc2652 (127.17.184.193:45527), d23ec306d46e48cea3ebf40d8ecfcc3f (127.17.184.195:44051)
I20260927 04:03:14.328807 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.325990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:55567 (local address 127.17.184.193:45527)
0927 04:03:14.326049 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.326051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.326061 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:14.326114 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.326115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.326124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.326198 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:14.326424 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.327035 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.328063 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.328482 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.328538 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.328544 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.328704 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.328731 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:03:14.328827 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.325902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:45527 (local address 127.17.184.194:55567)
0927 04:03:14.325964 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.325979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.326047 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.326240 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.326242 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.326243 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:14.326320 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.326322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.327115 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.327116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.327969 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.327972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.328557 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.328558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.328627 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.328631 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.328693 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.328732 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:03:14.329137 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.326142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:57219 (local address 127.17.184.195:44051)
0927 04:03:14.326189 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.326191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.326284 (+ 93us) server_negotiation.cc:407] Connection header received
0927 04:03:14.326336 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.326337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.326347 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.326402 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:14.326596 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.327231 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.328243 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.328745 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.328810 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.328817 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.329022 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.329065 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:03:14.329167 22376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b17325d760a4115b8fd82f544e66711" candidate_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" is_pre_election: true
I20260927 04:03:14.329336 22376 raft_consensus.cc:2454] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3384bf74f14e2e8a8649b7e98e7278 in term 0.
I20260927 04:03:14.329378 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.326024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44051 (local address 127.17.184.194:57219)
0927 04:03:14.326209 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.326219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.326274 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.326416 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.326416 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.326418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:14.326482 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.326484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.327317 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.327318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.328155 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:14.328158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.328843 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.328844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:14.328919 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.328923 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.329248 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.329306 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":121,"threads_started":1}
I20260927 04:03:14.329631 22443 leader_election.cc:302] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 268c7a0f7e334b2f8074b01bc0bc2652, bf3384bf74f14e2e8a8649b7e98e7278; no voters:
I20260927 04:03:14.329756 22712 raft_consensus.cc:2781] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:14.329810 22712 raft_consensus.cc:482] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:14.329839 22712 raft_consensus.cc:3037] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:14.329774 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b17325d760a4115b8fd82f544e66711" candidate_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" is_pre_election: true
I20260927 04:03:14.329941 22642 raft_consensus.cc:2454] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf3384bf74f14e2e8a8649b7e98e7278 in term 0.
I20260927 04:03:14.330629 22712 raft_consensus.cc:504] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.330760 22712 leader_election.cc:288] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [CANDIDATE]: Term 1 election: Requested vote from peers 268c7a0f7e334b2f8074b01bc0bc2652 (127.17.184.193:45527), d23ec306d46e48cea3ebf40d8ecfcc3f (127.17.184.195:44051)
I20260927 04:03:14.330932 22376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b17325d760a4115b8fd82f544e66711" candidate_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268c7a0f7e334b2f8074b01bc0bc2652"
I20260927 04:03:14.331014 22376 raft_consensus.cc:3037] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:14.330992 22642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b17325d760a4115b8fd82f544e66711" candidate_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f"
I20260927 04:03:14.331090 22642 raft_consensus.cc:3037] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:14.331727 22376 raft_consensus.cc:2454] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3384bf74f14e2e8a8649b7e98e7278 in term 1.
I20260927 04:03:14.331904 22642 raft_consensus.cc:2454] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf3384bf74f14e2e8a8649b7e98e7278 in term 1.
I20260927 04:03:14.331914 22443 leader_election.cc:302] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 268c7a0f7e334b2f8074b01bc0bc2652, bf3384bf74f14e2e8a8649b7e98e7278; no voters:
I20260927 04:03:14.332005 22712 raft_consensus.cc:2781] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:14.332191 22712 raft_consensus.cc:686] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 1 LEADER]: Becoming Leader. State: Replica: bf3384bf74f14e2e8a8649b7e98e7278, State: Running, Role: LEADER
I20260927 04:03:14.332326 22712 consensus_queue.cc:237] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } }
I20260927 04:03:14.333256 22232 catalog_manager.cc:5495] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf3384bf74f14e2e8a8649b7e98e7278 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3384bf74f14e2e8a8649b7e98e7278" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:14.384042 22702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.381522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.17.184.194:45943)
0927 04:03:14.381590 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.381592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.381601 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:14.381661 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.381662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.381680 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.381744 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:14.381884 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.382376 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.383243 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:14.383433 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:14.383489 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:14.383697 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:14.383799 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:14.383865 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.383906 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.383924 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:03:14.386022 22376 raft_consensus.cc:1261] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 1 FOLLOWER]: Refusing update from remote peer bf3384bf74f14e2e8a8649b7e98e7278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:14.386022 22642 raft_consensus.cc:1261] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 1 FOLLOWER]: Refusing update from remote peer bf3384bf74f14e2e8a8649b7e98e7278: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:14.386456 22715 consensus_queue.cc:1035] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d23ec306d46e48cea3ebf40d8ecfcc3f" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:14.386683 22712 consensus_queue.cc:1035] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "268c7a0f7e334b2f8074b01bc0bc2652" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:14.389163 22719 mvcc.cc:205] Tried to move back new op lower bound from 7333813429801766912 to 7333813429585784832. Current Snapshot: MvccSnapshot[applied={T|T < 7333813429801766912}]
I20260927 04:03:14.391462 22232 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:14.391556 22232 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:14.392397 22232 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdf56315e66c408292b19c69adb19e94: Sending DeleteTablet for 3 replicas of tablet 5b17325d760a4115b8fd82f544e66711
I20260927 04:03:14.392758 22356 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b17325d760a4115b8fd82f544e66711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:14 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:56816
I20260927 04:03:14.392766 22620 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b17325d760a4115b8fd82f544e66711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:14 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:57354
I20260927 04:03:14.393136 22729 tablet_replica.cc:330] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652: stopping tablet replica
I20260927 04:03:14.393127 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 22294
I20260927 04:03:14.393152 22489 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b17325d760a4115b8fd82f544e66711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:14 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:46392
I20260927 04:03:14.393231 22729 raft_consensus.cc:2229] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:14.393337 22729 raft_consensus.cc:2258] T 5b17325d760a4115b8fd82f544e66711 P 268c7a0f7e334b2f8074b01bc0bc2652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:14.394032 22728 tablet_replica.cc:330] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: stopping tablet replica
I20260927 04:03:14.394145 22728 raft_consensus.cc:2229] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:14.394243 22728 raft_consensus.cc:2258] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:14.394728 22728 ts_tablet_manager.cc:1905] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:14.396816 22728 ts_tablet_manager.cc:1918] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:14.396895 22728 log.cc:1196] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/5b17325d760a4115b8fd82f544e66711
I20260927 04:03:14.397208 22728 ts_tablet_manager.cc:1939] T 5b17325d760a4115b8fd82f544e66711 P d23ec306d46e48cea3ebf40d8ecfcc3f: Deleting consensus metadata
I20260927 04:03:14.397706 22217 catalog_manager.cc:4841] TS d23ec306d46e48cea3ebf40d8ecfcc3f (127.17.184.195:44051): tablet 5b17325d760a4115b8fd82f544e66711 (table test-table [id=5548d4d067d1465ebcea77de5fcf72e3]) successfully deleted
I20260927 04:03:14.397763 22730 tablet_replica.cc:330] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: stopping tablet replica
I20260927 04:03:14.397867 22730 raft_consensus.cc:2229] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:14.397972 22730 raft_consensus.cc:2258] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:14.398401 22730 ts_tablet_manager.cc:1905] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:14.400000 22730 ts_tablet_manager.cc:1918] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:14.400075 22730 log.cc:1196] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/5b17325d760a4115b8fd82f544e66711
I20260927 04:03:14.400331 22730 ts_tablet_manager.cc:1939] T 5b17325d760a4115b8fd82f544e66711 P bf3384bf74f14e2e8a8649b7e98e7278: Deleting consensus metadata
I20260927 04:03:14.400904 22218 catalog_manager.cc:4841] TS bf3384bf74f14e2e8a8649b7e98e7278 (127.17.184.194:45943): tablet 5b17325d760a4115b8fd82f544e66711 (table test-table [id=5548d4d067d1465ebcea77de5fcf72e3]) successfully deleted
W20260927 04:03:14.401072 22217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:45527 (error 108)
I20260927 04:03:14.401208 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 22427
I20260927 04:03:14.402876 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.402712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:45527 (local address 127.0.0.1:34342)
0927 04:03:14.402796 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.402845 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:45527: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 04:03:14.403069 22217 catalog_manager.cc:4584] TS 268c7a0f7e334b2f8074b01bc0bc2652 (127.17.184.193:45527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b17325d760a4115b8fd82f544e66711: Network error: Client connection negotiation failed: client connection to 127.17.184.193:45527: connect: Connection refused (error 111)
I20260927 04:03:14.405944 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 22560
I20260927 04:03:14.410410 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 22201
2026-09-27T04:03:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2163 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T04:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:14Z Disabled control of system clock
I20260927 04:03:14.437945 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:38383
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 04:03:14.518838 22735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:14.519073 22735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:14.519095 22735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:14.520251 22735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:14.520293 22735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:14.520306 22735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:14.520319 22735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:14.520371 22735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:14.521694 22735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:38383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:14.521965 22735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:14.522279 22735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:14.527102 22740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:14.527182 22741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:14.527179 22735 server_base.cc:1034] running on GCE node
W20260927 04:03:14.527132 22743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:14.527544 22735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:14.527818 22735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:14.528960 22735 hybrid_clock.cc:648] HybridClock initialized: now 1790481794528942 us; error 39 us; skew 500 ppm
I20260927 04:03:14.530261 22735 webserver.cc:460] Webserver started at http://127.17.184.254:35893/ using document root <none> and password file <none>
I20260927 04:03:14.530499 22735 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:14.530546 22735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:14.530694 22735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:14.531863 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "e7660e3796c9417b90186c43bf31df63"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "38e2a650ed33396bcc22b986cf249ea3"
server_key_iv: "9c31a8e1a7e6e643f39d067348d38e5c"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.532231 22735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "e7660e3796c9417b90186c43bf31df63"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "38e2a650ed33396bcc22b986cf249ea3"
server_key_iv: "9c31a8e1a7e6e643f39d067348d38e5c"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.533660 22735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:14.534539 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.534825 22735 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:14.534956 22735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "e7660e3796c9417b90186c43bf31df63"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "38e2a650ed33396bcc22b986cf249ea3"
server_key_iv: "9c31a8e1a7e6e643f39d067348d38e5c"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.535046 22735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:14.538376 22735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:14.538654 22735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:14.538806 22735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:14.542863 22735 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:38383
I20260927 04:03:14.543068 22801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:38383 every 8 connection(s)
I20260927 04:03:14.543359 22735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:14.544250 22802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:14.547331 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63: Bootstrap starting.
I20260927 04:03:14.547899 22802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:14.548156 22802 log.cc:824] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:14.548842 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63: No bootstrap required, opened a new log
I20260927 04:03:14.550513 22802 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7660e3796c9417b90186c43bf31df63" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38383 } }
I20260927 04:03:14.550661 22802 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:14.550686 22802 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7660e3796c9417b90186c43bf31df63, State: Initialized, Role: FOLLOWER
I20260927 04:03:14.550822 22802 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7660e3796c9417b90186c43bf31df63" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38383 } }
I20260927 04:03:14.550879 22802 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:14.550987 22802 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:14.551028 22802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:14.551577 22802 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7660e3796c9417b90186c43bf31df63" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38383 } }
I20260927 04:03:14.551667 22802 leader_election.cc:302] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7660e3796c9417b90186c43bf31df63; no voters:
I20260927 04:03:14.551801 22802 leader_election.cc:288] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:14.551877 22805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:14.552055 22805 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [term 1 LEADER]: Becoming Leader. State: Replica: e7660e3796c9417b90186c43bf31df63, State: Running, Role: LEADER
I20260927 04:03:14.552117 22802 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:14.552173 22805 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7660e3796c9417b90186c43bf31df63" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38383 } }
I20260927 04:03:14.552521 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 22735
I20260927 04:03:14.552625 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:14.552884 18147 external_mini_cluster.cc:1428] Setting key 12c88c7ac7191341e60893ace50eb489
I20260927 04:03:14.553720 22807 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7660e3796c9417b90186c43bf31df63. Latest consensus state: current_term: 1 leader_uuid: "e7660e3796c9417b90186c43bf31df63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7660e3796c9417b90186c43bf31df63" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38383 } } }
I20260927 04:03:14.553843 22807 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:14.554068 22806 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7660e3796c9417b90186c43bf31df63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7660e3796c9417b90186c43bf31df63" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38383 } } }
I20260927 04:03:14.554247 22806 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:14.554271 22823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:14.554800 22823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:14.555611 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.553391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50424 (local address 127.17.184.254:38383)
0927 04:03:14.553756 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.553765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.553800 (+ 35us) server_negotiation.cc:407] Connection header received
0927 04:03:14.553923 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.553931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.554064 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.554492 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.555152 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.555156 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.555156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:14.555160 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.555195 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:14.555205 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.555281 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.555364 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.555422 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":88,"threads_started":1}
I20260927 04:03:14.556569 22823 catalog_manager.cc:1347] Generated new cluster ID: 6e69b4bd9bc94e3e88f789864a26a9e4
I20260927 04:03:14.556622 22823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:14.569883 22823 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:14.570091 22823 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:14.578716 22823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63: Generated new TSK 0
I20260927 04:03:14.579011 22823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:03:14.584671 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:38383
--builtin_ntp_servers=127.17.184.212:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 04:03:14.675609 22826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:14.675865 22826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:14.675904 22826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:14.677120 22826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:14.677186 22826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:14.677232 22826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:14.678648 22826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:14.678977 22826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:14.679272 22826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:14.684441 22831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:14.684592 22834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:14.684454 22832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:14.684901 22826 server_base.cc:1034] running on GCE node
I20260927 04:03:14.685098 22826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:14.685364 22826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:14.686532 22826 hybrid_clock.cc:648] HybridClock initialized: now 1790481794686516 us; error 34 us; skew 500 ppm
I20260927 04:03:14.687912 22826 webserver.cc:460] Webserver started at http://127.17.184.193:43711/ using document root <none> and password file <none>
I20260927 04:03:14.688158 22826 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:14.688230 22826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:14.688396 22826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:14.689488 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "03d851057dfb45c484a8368369d418b8"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "dc9ff7e0ca9a538eda8a5b1377708237"
server_key_iv: "70aef749a0bf24f137dd4689a6371234"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.689863 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "03d851057dfb45c484a8368369d418b8"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "dc9ff7e0ca9a538eda8a5b1377708237"
server_key_iv: "70aef749a0bf24f137dd4689a6371234"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.691354 22826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:14.692250 22840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.692433 22826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:14.692538 22826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "03d851057dfb45c484a8368369d418b8"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "dc9ff7e0ca9a538eda8a5b1377708237"
server_key_iv: "70aef749a0bf24f137dd4689a6371234"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.692619 22826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:14.696143 22826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:14.696434 22826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:14.696583 22826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:14.696797 22826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:14.697111 22826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:14.697165 22826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.697203 22826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:14.697230 22826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.703693 22826 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:40885
I20260927 04:03:14.703742 22953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:40885 every 8 connection(s)
I20260927 04:03:14.704113 22826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:14.707779 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.704908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:43117 (local address 127.17.184.254:38383)
0927 04:03:14.704976 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.704978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.705250 (+ 272us) server_negotiation.cc:407] Connection header received
0927 04:03:14.705902 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.705903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.705909 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.705954 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.707367 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.707369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.707369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:14.707373 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.707390 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:14.707394 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.707475 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.707699 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.707722 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:03:14.708108 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.704899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:38383 (local address 127.17.184.193:43117)
0927 04:03:14.705099 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.705143 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.705795 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.706028 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.706032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.706043 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:14.707181 (+ 1138us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:14.707182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:14.707222 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:14.707268 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:14.707590 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:14.707596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:14.707599 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.707696 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.707765 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260927 04:03:14.708705 22954 heartbeater.cc:347] Connected to a master server at 127.17.184.254:38383
I20260927 04:03:14.708830 22954 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:14.709028 22954 heartbeater.cc:511] Master 127.17.184.254:38383 requested a full tablet report, sending...
I20260927 04:03:14.709578 22766 ts_manager.cc:194] Registered new tserver with Master: 03d851057dfb45c484a8368369d418b8 (127.17.184.193:40885)
I20260927 04:03:14.710580 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 22826
I20260927 04:03:14.710678 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:14.710871 18147 external_mini_cluster.cc:1428] Setting key f6b5ddcae0b079a4f0a071395d5aa81d
I20260927 04:03:14.711864 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:38383
--builtin_ntp_servers=127.17.184.212:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 04:03:14.795390 22957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:14.795600 22957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:14.795624 22957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:14.796877 22957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:14.796918 22957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:14.796944 22957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:14.798214 22957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:14.798419 22957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:14.798659 22957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:14.803673 22962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:14.803728 22963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:14.803718 22965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:14.804121 22957 server_base.cc:1034] running on GCE node
I20260927 04:03:14.804327 22957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:14.804525 22957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:14.805665 22957 hybrid_clock.cc:648] HybridClock initialized: now 1790481794805649 us; error 30 us; skew 500 ppm
I20260927 04:03:14.807120 22957 webserver.cc:460] Webserver started at http://127.17.184.194:44165/ using document root <none> and password file <none>
I20260927 04:03:14.807365 22957 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:14.807410 22957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:14.807593 22957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:14.808703 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "679c7b4a325d431e8b13e9f4a3d1472b"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "9c0a06d92db06b032b66af9e730b748f"
server_key_iv: "cd4ca98772e15dd54a46bbd8dd0f87ad"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.809005 22957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "679c7b4a325d431e8b13e9f4a3d1472b"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "9c0a06d92db06b032b66af9e730b748f"
server_key_iv: "cd4ca98772e15dd54a46bbd8dd0f87ad"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.810248 22957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:14.811082 22971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.811259 22957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:14.811350 22957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "679c7b4a325d431e8b13e9f4a3d1472b"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "9c0a06d92db06b032b66af9e730b748f"
server_key_iv: "cd4ca98772e15dd54a46bbd8dd0f87ad"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.811425 22957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:14.814735 22957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:14.815032 22957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:14.815196 22957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:14.815429 22957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:14.815733 22957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:14.815764 22957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.815819 22957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:14.815850 22957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.821708 22957 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:35657
I20260927 04:03:14.821848 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:35657 every 8 connection(s)
I20260927 04:03:14.822096 22957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:14.825451 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.823013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:35415 (local address 127.17.184.254:38383)
0927 04:03:14.823097 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.823099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.823317 (+ 218us) server_negotiation.cc:407] Connection header received
0927 04:03:14.823893 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.823895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.823902 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.823966 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.825239 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.825241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.825241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:14.825245 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.825259 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:14.825262 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.825317 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.825380 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.825391 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:03:14.825752 23087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.822976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:38383 (local address 127.17.184.194:35415)
0927 04:03:14.823183 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.823219 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.823733 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.824033 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.824037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.824045 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:14.825086 (+ 1041us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:14.825086 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:14.825116 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:14.825145 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:14.825318 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:14.825320 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:14.825323 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.825387 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.825436 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260927 04:03:14.826278 23085 heartbeater.cc:347] Connected to a master server at 127.17.184.254:38383
I20260927 04:03:14.826399 23085 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:14.826576 23085 heartbeater.cc:511] Master 127.17.184.254:38383 requested a full tablet report, sending...
I20260927 04:03:14.827015 22766 ts_manager.cc:194] Registered new tserver with Master: 679c7b4a325d431e8b13e9f4a3d1472b (127.17.184.194:35657)
I20260927 04:03:14.827105 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 22957
I20260927 04:03:14.827237 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:14.827459 18147 external_mini_cluster.cc:1428] Setting key b6202cf3079a4129014c85b459215ea5
I20260927 04:03:14.828326 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:38383
--builtin_ntp_servers=127.17.184.212:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 04:03:14.912904 23088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:14.913146 23088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:14.913172 23088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:14.914435 23088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:14.914476 23088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:14.914503 23088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:14.915944 23088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:38383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:14.916144 23088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:14.916385 23088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:14.921519 23094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:14.921530 23093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:14.921757 23088 server_base.cc:1034] running on GCE node
W20260927 04:03:14.921530 23096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:14.922024 23088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:14.922232 23088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:14.923367 23088 hybrid_clock.cc:648] HybridClock initialized: now 1790481794923327 us; error 60 us; skew 500 ppm
I20260927 04:03:14.924642 23088 webserver.cc:460] Webserver started at http://127.17.184.195:45287/ using document root <none> and password file <none>
I20260927 04:03:14.924913 23088 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:14.924963 23088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:14.925094 23088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:14.926177 23088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "56de716997c6419a9831b4f369aad3ca"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "bde28c581a84f3736e9661202004e381"
server_key_iv: "4c10e55aaf6a92329286c7b3de2e2ac9"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.926533 23088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "56de716997c6419a9831b4f369aad3ca"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "bde28c581a84f3736e9661202004e381"
server_key_iv: "4c10e55aaf6a92329286c7b3de2e2ac9"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.927943 23088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:14.928692 23102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.928885 23088 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:14.928987 23088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "56de716997c6419a9831b4f369aad3ca"
format_stamp: "Formatted at 2026-09-27 04:03:14 on dist-test-slave-zqrn"
server_key: "bde28c581a84f3736e9661202004e381"
server_key_iv: "4c10e55aaf6a92329286c7b3de2e2ac9"
server_key_version: "encryptionkey@0"
I20260927 04:03:14.929059 23088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:14.932533 23088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:14.932806 23088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:14.932940 23088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:14.933137 23088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:14.933437 23088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:14.933490 23088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.933528 23088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:14.933557 23088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:14.939517 23088 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:33457
I20260927 04:03:14.939568 23215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:33457 every 8 connection(s)
I20260927 04:03:14.939922 23088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:14.943048 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.940662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:60535 (local address 127.17.184.254:38383)
0927 04:03:14.940798 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.940801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.940992 (+ 191us) server_negotiation.cc:407] Connection header received
0927 04:03:14.941510 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.941511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.941517 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.941574 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.942737 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.942738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.942739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:14.942743 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.942759 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:14.942763 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.942837 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.942956 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.942974 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:03:14.943243 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.940657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:38383 (local address 127.17.184.195:60535)
0927 04:03:14.940876 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.940910 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.941403 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.941647 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.941651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.941661 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:14.942570 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:14.942570 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:14.942598 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:14.942624 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:14.942834 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:14.942840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:14.942843 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.942960 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.943022 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260927 04:03:14.943766 23216 heartbeater.cc:347] Connected to a master server at 127.17.184.254:38383
I20260927 04:03:14.943862 23216 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:14.944013 23216 heartbeater.cc:511] Master 127.17.184.254:38383 requested a full tablet report, sending...
I20260927 04:03:14.944034 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 23088
I20260927 04:03:14.944155 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:14.944351 18147 external_mini_cluster.cc:1428] Setting key 97c8a67230aed95944bc4b0a0a2ec9ab
I20260927 04:03:14.944381 22766 ts_manager.cc:194] Registered new tserver with Master: 56de716997c6419a9831b4f369aad3ca (127.17.184.195:33457)
I20260927 04:03:14.945169 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:03:14.947700 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.946493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50436 (local address 127.17.184.254:38383)
0927 04:03:14.946574 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.946576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.946667 (+ 91us) server_negotiation.cc:407] Connection header received
0927 04:03:14.946746 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.946747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.946753 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.946837 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.947477 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.947478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.947479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:14.947483 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.947498 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:14.947502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.947558 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.947615 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.947630 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 04:03:14.949319 22766 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:14.950801 22766 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:14.954978 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.953309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35657 (local address 127.0.0.1:54814)
0927 04:03:14.953531 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.953575 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.953644 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.953932 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.953938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.953944 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:14.954467 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:14.954468 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:14.954484 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:14.954510 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:14.954793 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:14.954799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:14.954803 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.954856 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.954873 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":60,"threads_started":1}
I20260927 04:03:14.955138 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.953693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.17.184.193:40885)
0927 04:03:14.953870 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.953874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.954061 (+ 187us) server_negotiation.cc:407] Connection header received
0927 04:03:14.954100 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.954103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.954113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.954413 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.954778 (+ 365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.954782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.954782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:14.954787 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.954816 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:14.954833 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.954922 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.955015 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.955033 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260927 04:03:14.955247 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.953399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.17.184.194:35657)
0927 04:03:14.953770 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.953775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.953786 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:14.953815 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.953818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.953828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.954166 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.954628 (+ 462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.954636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.954637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:14.954642 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.954681 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:14.954708 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.955103 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.955127 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.955145 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":111,"threads_started":1}
I20260927 04:03:14.955541 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.953600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:40885 (local address 127.0.0.1:47418)
0927 04:03:14.953935 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.953945 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.953995 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.954219 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.954220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.954222 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:14.954681 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:14.954682 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:14.954689 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:14.954702 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:14.954945 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:14.954946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:14.954947 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.955005 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.955019 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":72,"threads_started":1}
I20260927 04:03:14.955788 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.953321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:33457 (local address 127.0.0.1:40956)
0927 04:03:14.953832 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:14.953842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:14.953910 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:14.954976 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:14.954977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:14.954980 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:14.955391 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:14.955392 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:14.955400 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:14.955416 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:14.955663 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:14.955663 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:14.955664 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 04:03:14.955709 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 04:03:14.955721 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":136,"threads_started":1}
I20260927 04:03:14.955976 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:14.953608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.17.184.195:33457)
0927 04:03:14.953785 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:14.953789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:14.953891 (+ 102us) server_negotiation.cc:407] Connection header received
0927 04:03:14.954005 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:14.954009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:14.954018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:14.954676 (+ 658us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:14.955506 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:14.955512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:14.955513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:14.955517 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:14.955551 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:14.955577 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:14.955635 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:14.955879 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 04:03:14.955896 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":81,"threads_started":1}
I20260927 04:03:14.955999 23019 tablet_service.cc:1467] Processing CreateTablet for tablet f0c8c364e4574588a39d6cd589ea897e (DEFAULT_TABLE table=test-table [id=604577d60e6a43a1b47ef90c09f5a74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:14.956236 22887 tablet_service.cc:1467] Processing CreateTablet for tablet f0c8c364e4574588a39d6cd589ea897e (DEFAULT_TABLE table=test-table [id=604577d60e6a43a1b47ef90c09f5a74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:14.956430 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c8c364e4574588a39d6cd589ea897e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:14.956581 22887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c8c364e4574588a39d6cd589ea897e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:14.959260 23230 tablet_bootstrap.cc:492] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8: Bootstrap starting.
I20260927 04:03:14.959537 23231 tablet_bootstrap.cc:492] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Bootstrap starting.
I20260927 04:03:14.959594 23150 tablet_service.cc:1467] Processing CreateTablet for tablet f0c8c364e4574588a39d6cd589ea897e (DEFAULT_TABLE table=test-table [id=604577d60e6a43a1b47ef90c09f5a74c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:14.959931 23150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0c8c364e4574588a39d6cd589ea897e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:14.960073 23230 tablet_bootstrap.cc:654] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:14.960168 23231 tablet_bootstrap.cc:654] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:14.960408 23230 log.cc:824] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:14.960472 23231 log.cc:824] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:14.961211 23231 tablet_bootstrap.cc:492] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: No bootstrap required, opened a new log
I20260927 04:03:14.961212 23230 tablet_bootstrap.cc:492] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8: No bootstrap required, opened a new log
I20260927 04:03:14.961280 23230 ts_tablet_manager.cc:1397] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:14.961280 23231 ts_tablet_manager.cc:1397] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:14.962707 23234 tablet_bootstrap.cc:492] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: Bootstrap starting.
I20260927 04:03:14.963248 23234 tablet_bootstrap.cc:654] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:14.963462 23234 log.cc:824] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:14.964071 23234 tablet_bootstrap.cc:492] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: No bootstrap required, opened a new log
I20260927 04:03:14.964112 23234 ts_tablet_manager.cc:1397] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:14.964068 23230 raft_consensus.cc:348] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:14.964118 23231 raft_consensus.cc:348] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:14.964241 23231 raft_consensus.cc:374] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:14.964241 23230 raft_consensus.cc:374] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:14.964277 23231 raft_consensus.cc:729] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679c7b4a325d431e8b13e9f4a3d1472b, State: Initialized, Role: FOLLOWER
I20260927 04:03:14.964277 23230 raft_consensus.cc:729] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d851057dfb45c484a8368369d418b8, State: Initialized, Role: FOLLOWER
I20260927 04:03:14.964396 23230 consensus_queue.cc:260] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:14.964396 23231 consensus_queue.cc:260] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:14.964656 23230 ts_tablet_manager.cc:1428] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 04:03:14.964656 23231 ts_tablet_manager.cc:1428] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:03:14.964742 23085 heartbeater.cc:503] Master 127.17.184.254:38383 was elected leader, sending a full tablet report...
I20260927 04:03:14.964771 22954 heartbeater.cc:503] Master 127.17.184.254:38383 was elected leader, sending a full tablet report...
I20260927 04:03:14.966689 23234 raft_consensus.cc:348] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:14.966807 23234 raft_consensus.cc:374] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:14.966827 23234 raft_consensus.cc:729] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56de716997c6419a9831b4f369aad3ca, State: Initialized, Role: FOLLOWER
I20260927 04:03:14.966958 23234 consensus_queue.cc:260] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:14.967213 23234 ts_tablet_manager.cc:1428] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:03:14.967270 23216 heartbeater.cc:503] Master 127.17.184.254:38383 was elected leader, sending a full tablet report...
W20260927 04:03:15.072747 23086 tablet.cc:2369] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:15.171890 23238 raft_consensus.cc:482] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:15.172034 23238 raft_consensus.cc:504] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:15.172492 23238 leader_election.cc:288] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 679c7b4a325d431e8b13e9f4a3d1472b (127.17.184.194:35657), 03d851057dfb45c484a8368369d418b8 (127.17.184.193:40885)
I20260927 04:03:15.173905 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.172742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:40885 (local address 127.17.184.195:56713)
0927 04:03:15.172841 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:15.172855 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:15.172919 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:15.173093 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:15.173094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:15.173097 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:15.173535 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:15.173536 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:15.173544 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:15.173557 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:15.173780 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:15.173781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:15.173782 (+ 1us) client_negotiation.cc:769] Sending connection context
0927 04:03:15.173843 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 04:03:15.173856 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 04:03:15.174049 23229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.172845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:56713 (local address 127.17.184.193:40885)
0927 04:03:15.172917 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.172919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.172932 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:03:15.172997 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.172998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.173005 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.173074 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:15.173698 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:15.173699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:15.173699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:15.173703 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:15.173717 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:15.173720 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:15.173764 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:15.173962 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 04:03:15.173977 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:03:15.174422 22889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c8c364e4574588a39d6cd589ea897e" candidate_uuid: "56de716997c6419a9831b4f369aad3ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d851057dfb45c484a8368369d418b8" is_pre_election: true
I20260927 04:03:15.174597 22889 raft_consensus.cc:2454] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56de716997c6419a9831b4f369aad3ca in term 0.
I20260927 04:03:15.174616 23239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.172741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:35657 (local address 127.17.184.195:60825)
0927 04:03:15.173101 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:15.173113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:15.173206 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:15.173625 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:15.173626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:15.173630 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:15.174252 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:15.174252 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:15.174258 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:15.174270 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:15.174462 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:15.174463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:15.174463 (+ 0us) client_negotiation.cc:769] Sending connection context
0927 04:03:15.174501 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 04:03:15.174511 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":151,"thread_start_us":109,"threads_started":1}
I20260927 04:03:15.174577 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.172853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:60825 (local address 127.17.184.194:35657)
0927 04:03:15.172903 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.172905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.173210 (+ 305us) server_negotiation.cc:407] Connection header received
0927 04:03:15.173525 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.173526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.173531 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.173580 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:15.174355 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:15.174357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:15.174357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:15.174362 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:15.174378 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:15.174382 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:15.174462 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:15.174502 (+ 40us) server_negotiation.cc:299] Negotiation successful
0927 04:03:15.174516 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 04:03:15.174849 23103 leader_election.cc:302] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d851057dfb45c484a8368369d418b8, 56de716997c6419a9831b4f369aad3ca; no voters:
I20260927 04:03:15.174988 23238 raft_consensus.cc:2781] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:15.175041 23238 raft_consensus.cc:482] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:15.175099 23238 raft_consensus.cc:3037] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:15.175061 23039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c8c364e4574588a39d6cd589ea897e" candidate_uuid: "56de716997c6419a9831b4f369aad3ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" is_pre_election: true
I20260927 04:03:15.175236 23039 raft_consensus.cc:2454] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56de716997c6419a9831b4f369aad3ca in term 0.
I20260927 04:03:15.175880 23238 raft_consensus.cc:504] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:15.176021 23238 leader_election.cc:288] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [CANDIDATE]: Term 1 election: Requested vote from peers 679c7b4a325d431e8b13e9f4a3d1472b (127.17.184.194:35657), 03d851057dfb45c484a8368369d418b8 (127.17.184.193:40885)
I20260927 04:03:15.176129 23039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c8c364e4574588a39d6cd589ea897e" candidate_uuid: "56de716997c6419a9831b4f369aad3ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "679c7b4a325d431e8b13e9f4a3d1472b"
I20260927 04:03:15.176146 22889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0c8c364e4574588a39d6cd589ea897e" candidate_uuid: "56de716997c6419a9831b4f369aad3ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d851057dfb45c484a8368369d418b8"
I20260927 04:03:15.176213 22889 raft_consensus.cc:3037] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:15.176229 23039 raft_consensus.cc:3037] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:15.176918 22889 raft_consensus.cc:2454] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56de716997c6419a9831b4f369aad3ca in term 1.
I20260927 04:03:15.177012 23039 raft_consensus.cc:2454] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56de716997c6419a9831b4f369aad3ca in term 1.
I20260927 04:03:15.177078 23103 leader_election.cc:302] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d851057dfb45c484a8368369d418b8, 56de716997c6419a9831b4f369aad3ca; no voters:
I20260927 04:03:15.177153 23238 raft_consensus.cc:2781] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:15.177284 23238 raft_consensus.cc:686] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 1 LEADER]: Becoming Leader. State: Replica: 56de716997c6419a9831b4f369aad3ca, State: Running, Role: LEADER
I20260927 04:03:15.177382 23238 consensus_queue.cc:237] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } }
I20260927 04:03:15.178354 22766 catalog_manager.cc:5495] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 56de716997c6419a9831b4f369aad3ca (127.17.184.195). New cstate: current_term: 1 leader_uuid: "56de716997c6419a9831b4f369aad3ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56de716997c6419a9831b4f369aad3ca" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 33457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:15.183840 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.182872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.17.184.195:33457)
0927 04:03:15.182947 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.182949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.182957 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:15.182999 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.183000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.183010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.183060 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:15.183545 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:15.183547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:15.183547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:15.183551 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:15.183564 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:15.183568 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:15.183620 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:15.183670 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 04:03:15.183681 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:03:15.185541 23039 raft_consensus.cc:1261] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 1 FOLLOWER]: Refusing update from remote peer 56de716997c6419a9831b4f369aad3ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:15.185572 22889 raft_consensus.cc:1261] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 1 FOLLOWER]: Refusing update from remote peer 56de716997c6419a9831b4f369aad3ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:15.185854 23240 consensus_queue.cc:1035] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d851057dfb45c484a8368369d418b8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 40885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:15.185947 23238 consensus_queue.cc:1035] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "679c7b4a325d431e8b13e9f4a3d1472b" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 35657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:15.188329 23244 mvcc.cc:205] Tried to move back new op lower bound from 7333813433077280768 to 7333813433047285760. Current Snapshot: MvccSnapshot[applied={T|T < 7333813433077280768}]
I20260927 04:03:15.190709 22765 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:15.190809 22765 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:15.191639 22765 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7660e3796c9417b90186c43bf31df63: Sending DeleteTablet for 3 replicas of tablet f0c8c364e4574588a39d6cd589ea897e
I20260927 04:03:15.191987 23019 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c8c364e4574588a39d6cd589ea897e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:15 UTC) from {username='slave'} at 127.0.0.1:54814
I20260927 04:03:15.192044 22887 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c8c364e4574588a39d6cd589ea897e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:15 UTC) from {username='slave'} at 127.0.0.1:47418
I20260927 04:03:15.192035 23150 tablet_service.cc:1514] Processing DeleteTablet for tablet f0c8c364e4574588a39d6cd589ea897e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:15 UTC) from {username='slave'} at 127.0.0.1:40956
I20260927 04:03:15.192358 23254 tablet_replica.cc:330] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8: stopping tablet replica
I20260927 04:03:15.192466 23254 raft_consensus.cc:2229] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:15.192570 23254 raft_consensus.cc:2258] T f0c8c364e4574588a39d6cd589ea897e P 03d851057dfb45c484a8368369d418b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:15.192615 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 22826
I20260927 04:03:15.194087 23253 tablet_replica.cc:330] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: stopping tablet replica
I20260927 04:03:15.194164 23253 raft_consensus.cc:2229] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:15.194239 23253 raft_consensus.cc:2258] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:15.194409 23252 tablet_replica.cc:330] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: stopping tablet replica
I20260927 04:03:15.194502 23252 raft_consensus.cc:2229] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:15.194593 23252 raft_consensus.cc:2258] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:15.194803 23253 ts_tablet_manager.cc:1905] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:15.195000 23252 ts_tablet_manager.cc:1905] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:15.197049 23253 ts_tablet_manager.cc:1918] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:15.197049 23252 ts_tablet_manager.cc:1918] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:15.197129 23252 log.cc:1196] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/f0c8c364e4574588a39d6cd589ea897e
I20260927 04:03:15.197146 23253 log.cc:1196] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/f0c8c364e4574588a39d6cd589ea897e
I20260927 04:03:15.197396 23252 ts_tablet_manager.cc:1939] T f0c8c364e4574588a39d6cd589ea897e P 679c7b4a325d431e8b13e9f4a3d1472b: Deleting consensus metadata
I20260927 04:03:15.197407 23253 ts_tablet_manager.cc:1939] T f0c8c364e4574588a39d6cd589ea897e P 56de716997c6419a9831b4f369aad3ca: Deleting consensus metadata
I20260927 04:03:15.197808 22753 catalog_manager.cc:4841] TS 679c7b4a325d431e8b13e9f4a3d1472b (127.17.184.194:35657): tablet f0c8c364e4574588a39d6cd589ea897e (table test-table [id=604577d60e6a43a1b47ef90c09f5a74c]) successfully deleted
I20260927 04:03:15.197917 22750 catalog_manager.cc:4841] TS 56de716997c6419a9831b4f369aad3ca (127.17.184.195:33457): tablet f0c8c364e4574588a39d6cd589ea897e (table test-table [id=604577d60e6a43a1b47ef90c09f5a74c]) successfully deleted
W20260927 04:03:15.199373 22750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:40885 (error 108)
I20260927 04:03:15.199769 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 22957
I20260927 04:03:15.199965 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.199796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:40885 (local address 127.0.0.1:47420)
0927 04:03:15.199872 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:15.199922 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:40885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 04:03:15.200450 22750 catalog_manager.cc:4584] TS 03d851057dfb45c484a8368369d418b8 (127.17.184.193:40885): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0c8c364e4574588a39d6cd589ea897e: Network error: Client connection negotiation failed: client connection to 127.17.184.193:40885: connect: Connection refused (error 111)
I20260927 04:03:15.207017 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 23088
I20260927 04:03:15.213346 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 22735
2026-09-27T04:03:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (802 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 04:03:15.233701 18147 mini_oidc.cc:152] Starting JWKS server
I20260927 04:03:15.234809 18147 webserver.cc:460] Webserver started at https://127.0.0.1:33743/ using document root <none> and password file <none>
I20260927 04:03:15.234954 18147 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 04:03:15.235167 18147 webserver.cc:460] Webserver started at http://127.0.0.1:35757/ using document root <none> and password file <none>
2026-09-27T04:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:15Z Disabled control of system clock
I20260927 04:03:15.243752 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:42935
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:36647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:42935
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33743/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/testchainca.cert with env {}
W20260927 04:03:15.334404 23261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:15.334662 23261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:15.334729 23261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:15.336185 23261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:15.336236 23261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:15.336251 23261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:15.336264 23261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:15.336278 23261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:03:15.336292 23261 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 04:03:15.336304 23261 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33743/jwks.json
I20260927 04:03:15.338004 23261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:36647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:42935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:42935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33743/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:15.338294 23261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:15.338536 23261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:15.343891 23266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:15.343870 23267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:15.344028 23261 server_base.cc:1034] running on GCE node
W20260927 04:03:15.343892 23269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:15.344334 23261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:15.344691 23261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:15.345883 23261 hybrid_clock.cc:648] HybridClock initialized: now 1790481795345848 us; error 46 us; skew 500 ppm
I20260927 04:03:15.347505 23261 webserver.cc:460] Webserver started at http://127.17.184.254:33507/ using document root <none> and password file <none>
I20260927 04:03:15.347862 23261 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:15.347951 23261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:15.348119 23261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:15.349414 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "8e2785cf51e846c9ab5180173d44d3c5"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "5569405f73db55177c099092d519c1d8"
server_key_iv: "e8b1f0e97b66e224443a53e9e16bbc83"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.349843 23261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "8e2785cf51e846c9ab5180173d44d3c5"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "5569405f73db55177c099092d519c1d8"
server_key_iv: "e8b1f0e97b66e224443a53e9e16bbc83"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.351320 23261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260927 04:03:15.352242 23275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:15.352445 23261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:15.352561 23261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "8e2785cf51e846c9ab5180173d44d3c5"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "5569405f73db55177c099092d519c1d8"
server_key_iv: "e8b1f0e97b66e224443a53e9e16bbc83"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.352643 23261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:15.364435 23261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:15.364766 23261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:15.364923 23261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:15.368741 23261 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:42935
I20260927 04:03:15.368784 23327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:42935 every 8 connection(s)
I20260927 04:03:15.369134 23261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:15.370116 23328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:15.372589 23328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5: Bootstrap starting.
I20260927 04:03:15.373221 23328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:15.373526 23328 log.cc:824] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:15.374274 23328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5: No bootstrap required, opened a new log
I20260927 04:03:15.375993 23328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2785cf51e846c9ab5180173d44d3c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42935 } }
I20260927 04:03:15.376133 23328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:15.376156 23328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2785cf51e846c9ab5180173d44d3c5, State: Initialized, Role: FOLLOWER
I20260927 04:03:15.376309 23328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2785cf51e846c9ab5180173d44d3c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42935 } }
I20260927 04:03:15.376385 23328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:15.376407 23328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:15.376477 23328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:15.377007 23328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2785cf51e846c9ab5180173d44d3c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42935 } }
I20260927 04:03:15.377122 23328 leader_election.cc:302] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2785cf51e846c9ab5180173d44d3c5; no voters:
I20260927 04:03:15.377319 23328 leader_election.cc:288] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:15.377413 23331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:15.377566 23328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:15.377573 23331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8e2785cf51e846c9ab5180173d44d3c5, State: Running, Role: LEADER
I20260927 04:03:15.377734 23331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2785cf51e846c9ab5180173d44d3c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42935 } }
I20260927 04:03:15.378199 23333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e2785cf51e846c9ab5180173d44d3c5. Latest consensus state: current_term: 1 leader_uuid: "8e2785cf51e846c9ab5180173d44d3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2785cf51e846c9ab5180173d44d3c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42935 } } }
I20260927 04:03:15.378309 23333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:15.378582 23340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:15.378294 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e2785cf51e846c9ab5180173d44d3c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2785cf51e846c9ab5180173d44d3c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42935 } } }
I20260927 04:03:15.378844 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:15.378875 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 23261
I20260927 04:03:15.379006 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:15.379227 18147 external_mini_cluster.cc:1428] Setting key 7f436a7559f17f3d5623bab8ff33ebf2
I20260927 04:03:15.379869 23340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:15.381400 23340 catalog_manager.cc:1347] Generated new cluster ID: c4870dba29024948bbb1b69d3c76fdfb
I20260927 04:03:15.381460 23340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:15.385353 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.379706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.17.184.254:42935)
0927 04:03:15.379936 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.379946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.379985 (+ 39us) server_negotiation.cc:407] Connection header received
0927 04:03:15.380110 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.380118 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.380205 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.380364 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:15.380538 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.382045 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.382887 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.383582 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.383705 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.384481 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:15.384485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:15.384486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:15.384712 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:15.384760 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:15.384775 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:15.384847 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:15.384939 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 04:03:15.385118 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260927 04:03:15.386303 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:03:15.395752 23340 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:15.396353 23340 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:15.412846 23340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e2785cf51e846c9ab5180173d44d3c5: Generated new TSK 0
I20260927 04:03:15.413061 23340 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 04:03:15.443004 23358 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 04:03:15.443135 23358 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 04:03:15.443393 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.427516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.17.184.254:42935)
0927 04:03:15.427569 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.427571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.427684 (+ 113us) server_negotiation.cc:407] Connection header received
0927 04:03:15.427797 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.427798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.427820 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.427877 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0927 04:03:15.428031 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.428517 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.429797 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.429972 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.430017 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.430298 (+ 281us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 04:03:15.443134 (+ 12836us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 04:03:15.443336 (+ 202us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46432: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 04:03:15.443473 23350 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46432: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 04:03:15.443874 23357 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:15.427408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42935 (local address 127.0.0.1:46432)
0927 04:03:15.427604 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:15.427618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:15.427697 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:15.427887 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:15.427888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:15.427891 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0927 04:03:15.427963 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:15.427965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.428622 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.428623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:15.429703 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:15.429706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.430058 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.430059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:15.430130 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.430161 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 04:03:15.443363 (+ 13202us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 04:03:15.443516 (+ 153us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.184.254:42935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.17.184.254:42935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33743/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 04:03:15.444563 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 23261
2026-09-27T04:03:15Z chronyd exiting
I20260927 04:03:15.481276 18147 test_util.cc:175] -----------------------------------------------
I20260927 04:03:15.481362 18147 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790481773036938-18147-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (248 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 04:03:15.482196 18147 mini_oidc.cc:152] Starting JWKS server
I20260927 04:03:15.482939 18147 webserver.cc:460] Webserver started at https://127.0.0.1:43763/ using document root <none> and password file <none>
I20260927 04:03:15.482982 18147 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 04:03:15.483162 18147 webserver.cc:460] Webserver started at http://127.0.0.1:43647/ using document root <none> and password file <none>
2026-09-27T04:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:15Z Disabled control of system clock
I20260927 04:03:15.491747 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:46853
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:39549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:46853
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43763/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/testchainca.cert with env {}
W20260927 04:03:15.573401 23364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:15.573603 23364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:15.573624 23364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:15.574720 23364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:15.574756 23364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:15.574769 23364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:15.574781 23364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:15.574793 23364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:03:15.574805 23364 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 04:03:15.574816 23364 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43763/jwks.json
I20260927 04:03:15.576435 23364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:39549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:46853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:46853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43763/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:15.576635 23364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:15.576802 23364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:15.581956 23369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:15.581992 23370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:15.582038 23364 server_base.cc:1034] running on GCE node
W20260927 04:03:15.582003 23372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:15.582597 23364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:15.583029 23364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:15.584232 23364 hybrid_clock.cc:648] HybridClock initialized: now 1790481795584195 us; error 39 us; skew 500 ppm
I20260927 04:03:15.585712 23364 webserver.cc:460] Webserver started at http://127.17.184.254:40431/ using document root <none> and password file <none>
I20260927 04:03:15.586004 23364 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:15.586076 23364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:15.586215 23364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:15.587419 23364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "13f244fe37194ba1821ea6d7b721a396"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "8304c45dbf3d88a343b7fa0908d073db"
server_key_iv: "5936c48cf24ee3241080d65e4fc06a9c"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.587790 23364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "13f244fe37194ba1821ea6d7b721a396"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "8304c45dbf3d88a343b7fa0908d073db"
server_key_iv: "5936c48cf24ee3241080d65e4fc06a9c"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.589126 23364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:15.589950 23378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:15.590200 23364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:15.590325 23364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "13f244fe37194ba1821ea6d7b721a396"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "8304c45dbf3d88a343b7fa0908d073db"
server_key_iv: "5936c48cf24ee3241080d65e4fc06a9c"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.590415 23364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:15.606537 23364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:15.606976 23364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:15.607144 23364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:15.611646 23364 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:46853
I20260927 04:03:15.611718 23430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:46853 every 8 connection(s)
I20260927 04:03:15.612102 23364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:15.612865 23431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:15.615860 23431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396: Bootstrap starting.
I20260927 04:03:15.616377 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 23364
I20260927 04:03:15.616431 23431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:15.616508 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:15.616724 18147 external_mini_cluster.cc:1428] Setting key a92eee779517a289699dd02322fa59f1
I20260927 04:03:15.616765 23431 log.cc:824] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:15.617754 23431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396: No bootstrap required, opened a new log
I20260927 04:03:15.620603 23431 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f244fe37194ba1821ea6d7b721a396" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46853 } }
I20260927 04:03:15.620759 23431 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:15.620783 23431 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f244fe37194ba1821ea6d7b721a396, State: Initialized, Role: FOLLOWER
I20260927 04:03:15.620937 23431 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f244fe37194ba1821ea6d7b721a396" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46853 } }
I20260927 04:03:15.621012 23431 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:15.621060 23431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:15.621112 23431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:15.621928 23431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f244fe37194ba1821ea6d7b721a396" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46853 } }
I20260927 04:03:15.622046 23431 leader_election.cc:302] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13f244fe37194ba1821ea6d7b721a396; no voters:
I20260927 04:03:15.622236 23431 leader_election.cc:288] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:15.622274 23436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:15.622460 23436 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [term 1 LEADER]: Becoming Leader. State: Replica: 13f244fe37194ba1821ea6d7b721a396, State: Running, Role: LEADER
I20260927 04:03:15.622586 23436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f244fe37194ba1821ea6d7b721a396" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46853 } }
I20260927 04:03:15.622587 23431 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:15.622990 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13f244fe37194ba1821ea6d7b721a396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f244fe37194ba1821ea6d7b721a396" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46853 } } }
I20260927 04:03:15.622982 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13f244fe37194ba1821ea6d7b721a396. Latest consensus state: current_term: 1 leader_uuid: "13f244fe37194ba1821ea6d7b721a396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13f244fe37194ba1821ea6d7b721a396" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46853 } } }
I20260927 04:03:15.623077 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:15.623085 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:15.623297 23440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:15.624087 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.617261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.17.184.254:46853)
0927 04:03:15.617446 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.617456 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.617492 (+ 36us) server_negotiation.cc:407] Connection header received
0927 04:03:15.618170 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.618179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.618277 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.618448 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:15.618608 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.620297 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.621343 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.622022 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.622158 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.622980 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:15.622985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:15.622986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:15.623218 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:15.623284 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:15.623304 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:15.623690 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:15.623728 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 04:03:15.623820 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260927 04:03:15.623831 23440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:15.625824 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:03:15.626307 23440 catalog_manager.cc:1347] Generated new cluster ID: b721dab4e8774f05b3b3633e6f457497
I20260927 04:03:15.626355 23440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:15.650282 23440 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:15.651023 23440 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:15.658279 23440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13f244fe37194ba1821ea6d7b721a396: Generated new TSK 0
I20260927 04:03:15.658502 23440 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 04:03:15.690296 23461 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 04:03:15.690390 23461 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 04:03:15.690687 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.668246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36262 (local address 127.17.184.254:46853)
0927 04:03:15.668318 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.668320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.668407 (+ 87us) server_negotiation.cc:407] Connection header received
0927 04:03:15.668526 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.668527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.668539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.668681 (+ 142us) server_negotiation.cc:226] Negotiated authn=JWT
0927 04:03:15.668846 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.669385 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.670220 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.670425 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.670517 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.670727 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 04:03:15.690434 (+ 19707us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 04:03:15.690613 (+ 179us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 04:03:15.690754 23435 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36262: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 04:03:15.690845 23460 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:15.668167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:46853 (local address 127.0.0.1:36262)
0927 04:03:15.668323 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:15.668336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:15.668428 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:15.668693 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:15.668694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:15.668696 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 04:03:15.668770 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:15.668773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.669478 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.669479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:15.670106 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:15.670109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.670522 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.670523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:15.670583 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.670591 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 04:03:15.690610 (+ 20019us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 04:03:15.690735 (+ 125us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.184.254:46853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260927 04:03:15.691342 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 23364
2026-09-27T04:03:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (246 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 04:03:15.728643 18147 mini_oidc.cc:152] Starting JWKS server
I20260927 04:03:15.729316 18147 webserver.cc:460] Webserver started at https://127.0.0.1:40499/ using document root <none> and password file <none>
I20260927 04:03:15.729360 18147 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 04:03:15.729506 18147 webserver.cc:460] Webserver started at http://127.0.0.1:33391/ using document root <none> and password file <none>
2026-09-27T04:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:15Z Disabled control of system clock
I20260927 04:03:15.737501 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:35301
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:35301
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40499/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 04:03:15.817848 23467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:15.818109 23467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:15.818142 23467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:15.819334 23467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:15.819413 23467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:15.819433 23467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:15.819453 23467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:15.819485 23467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:03:15.819509 23467 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 04:03:15.819527 23467 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40499/jwks.json
I20260927 04:03:15.820914 23467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:35301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:35301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40499/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:15.821198 23467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:15.821491 23467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:15.826673 23475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:15.826673 23473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:15.826733 23472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:15.826835 23467 server_base.cc:1034] running on GCE node
I20260927 04:03:15.827184 23467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:15.827481 23467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:15.828857 23467 hybrid_clock.cc:648] HybridClock initialized: now 1790481795828845 us; error 36 us; skew 500 ppm
I20260927 04:03:15.830353 23467 webserver.cc:460] Webserver started at http://127.17.184.254:33481/ using document root <none> and password file <none>
I20260927 04:03:15.830616 23467 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:15.830693 23467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:15.830862 23467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:15.832204 23467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "3769be4f46e5467f945f70a220d7e4bc"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "d23daad3e3d493d582227ff89127772a"
server_key_iv: "bbf498247b6136229057ca0f34555d7e"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.832615 23467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "3769be4f46e5467f945f70a220d7e4bc"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "d23daad3e3d493d582227ff89127772a"
server_key_iv: "bbf498247b6136229057ca0f34555d7e"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.834034 23467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:03:15.834947 23481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:15.835150 23467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:15.835258 23467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "3769be4f46e5467f945f70a220d7e4bc"
format_stamp: "Formatted at 2026-09-27 04:03:15 on dist-test-slave-zqrn"
server_key: "d23daad3e3d493d582227ff89127772a"
server_key_iv: "bbf498247b6136229057ca0f34555d7e"
server_key_version: "encryptionkey@0"
I20260927 04:03:15.835350 23467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:15.848799 23467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:15.849159 23467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:15.849341 23467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:15.853688 23467 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:35301
I20260927 04:03:15.853772 23533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:35301 every 8 connection(s)
I20260927 04:03:15.854131 23467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:15.854861 23534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:15.857717 23534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc: Bootstrap starting.
I20260927 04:03:15.858318 23534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:15.858613 23534 log.cc:824] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:15.859453 23534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc: No bootstrap required, opened a new log
I20260927 04:03:15.861166 23534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3769be4f46e5467f945f70a220d7e4bc" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 35301 } }
I20260927 04:03:15.861339 23534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:15.861421 23534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3769be4f46e5467f945f70a220d7e4bc, State: Initialized, Role: FOLLOWER
I20260927 04:03:15.861618 23534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3769be4f46e5467f945f70a220d7e4bc" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 35301 } }
I20260927 04:03:15.861727 23534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:15.861775 23534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:15.861833 23534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:15.862010 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 23467
I20260927 04:03:15.862097 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:15.862282 18147 external_mini_cluster.cc:1428] Setting key f81780f9c9feb9ffa80855d2bb0d5d00
I20260927 04:03:15.862578 23534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3769be4f46e5467f945f70a220d7e4bc" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 35301 } }
I20260927 04:03:15.862715 23534 leader_election.cc:302] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3769be4f46e5467f945f70a220d7e4bc; no voters:
I20260927 04:03:15.862877 23534 leader_election.cc:288] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:15.862927 23538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:15.863121 23538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [term 1 LEADER]: Becoming Leader. State: Replica: 3769be4f46e5467f945f70a220d7e4bc, State: Running, Role: LEADER
I20260927 04:03:15.863175 23534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:15.863272 23538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3769be4f46e5467f945f70a220d7e4bc" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 35301 } }
I20260927 04:03:15.863677 23538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3769be4f46e5467f945f70a220d7e4bc. Latest consensus state: current_term: 1 leader_uuid: "3769be4f46e5467f945f70a220d7e4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3769be4f46e5467f945f70a220d7e4bc" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 35301 } } }
I20260927 04:03:15.863699 23540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3769be4f46e5467f945f70a220d7e4bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3769be4f46e5467f945f70a220d7e4bc" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 35301 } } }
I20260927 04:03:15.863775 23538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:15.863780 23540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:15.864066 23547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:15.864701 23547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:15.866544 23547 catalog_manager.cc:1347] Generated new cluster ID: 05fdeadb331c416ca540f2d0944d1a8f
I20260927 04:03:15.866669 23547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:15.870484 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.862794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.17.184.254:35301)
0927 04:03:15.864056 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.864067 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.864101 (+ 34us) server_negotiation.cc:407] Connection header received
0927 04:03:15.864473 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.864484 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.864583 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.865005 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:15.865119 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.866761 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.867943 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.868657 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.868797 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.869587 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:15.869591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:15.869592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:15.869820 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:15.869865 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:15.869884 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:15.869963 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:15.870036 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 04:03:15.870249 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":73,"threads_started":1}
I20260927 04:03:15.871480 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:03:15.893180 23547 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:15.893951 23547 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:15.903146 23547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3769be4f46e5467f945f70a220d7e4bc: Generated new TSK 0
I20260927 04:03:15.903390 23547 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:03:15.913661 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:15.909735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.17.184.254:35301)
0927 04:03:15.909785 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:15.909787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:15.909865 (+ 78us) server_negotiation.cc:407] Connection header received
0927 04:03:15.909910 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:15.909911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:15.909924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:15.910020 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0927 04:03:15.910138 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.910963 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.911835 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.912152 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.912225 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.912415 (+ 190us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 04:03:15.913425 (+ 1010us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 04:03:15.913570 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50650: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":27}
W20260927 04:03:15.913738 23539 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50650: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 04:03:15.913970 23563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:15.909623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:35301 (local address 127.0.0.1:50650)
0927 04:03:15.909780 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:15.909792 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:15.909858 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:15.910018 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:15.910019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:15.910020 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0927 04:03:15.910067 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:15.910069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.911097 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.911099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:15.911736 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:15.911738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:15.912239 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:15.912240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:15.912285 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:15.912291 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 04:03:15.913533 (+ 1242us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 04:03:15.913645 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.184.254:35301: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40499/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":110,"threads_started":1}
I20260927 04:03:15.914273 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 23467
E20260927 04:03:15.927742 23564 webserver.cc:533] Webserver: sslize failed:
E20260927 04:03:15.927826 23564 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T04:03:15Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T04:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:15Z Disabled control of system clock
I20260927 04:03:15.961258 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:45071
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:39225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:45071
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/testchainca.cert with env {}
W20260927 04:03:16.042111 23568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:16.042330 23568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:16.042352 23568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:16.043502 23568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:16.043536 23568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:16.043550 23568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:16.043561 23568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:16.043573 23568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 04:03:16.043586 23568 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 04:03:16.043597 23568 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 04:03:16.045038 23568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:39225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:45071
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:45071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:16.045238 23568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:16.045410 23568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:16.050513 23573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:16.050556 23576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:16.050556 23574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:16.050760 23568 server_base.cc:1034] running on GCE node
I20260927 04:03:16.051047 23568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:16.051347 23568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:16.052515 23568 hybrid_clock.cc:648] HybridClock initialized: now 1790481796052497 us; error 40 us; skew 500 ppm
I20260927 04:03:16.053944 23568 webserver.cc:460] Webserver started at http://127.17.184.254:32831/ using document root <none> and password file <none>
I20260927 04:03:16.054243 23568 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:16.054306 23568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:16.054440 23568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:16.055631 23568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "ec2e34831c424b4788784017dea19392"
format_stamp: "Formatted at 2026-09-27 04:03:16 on dist-test-slave-zqrn"
server_key: "43da11a4990b466af28f3c49b55e938c"
server_key_iv: "7cc7a5ad39d1e0aa96ceaaa23005febd"
server_key_version: "encryptionkey@0"
I20260927 04:03:16.056044 23568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "ec2e34831c424b4788784017dea19392"
format_stamp: "Formatted at 2026-09-27 04:03:16 on dist-test-slave-zqrn"
server_key: "43da11a4990b466af28f3c49b55e938c"
server_key_iv: "7cc7a5ad39d1e0aa96ceaaa23005febd"
server_key_version: "encryptionkey@0"
I20260927 04:03:16.057402 23568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:16.058271 23582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:16.058547 23568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:16.058710 23568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "ec2e34831c424b4788784017dea19392"
format_stamp: "Formatted at 2026-09-27 04:03:16 on dist-test-slave-zqrn"
server_key: "43da11a4990b466af28f3c49b55e938c"
server_key_iv: "7cc7a5ad39d1e0aa96ceaaa23005febd"
server_key_version: "encryptionkey@0"
I20260927 04:03:16.058842 23568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:16.067044 23568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:16.067358 23568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:16.067507 23568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:16.071940 23568 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:45071
I20260927 04:03:16.071985 23634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:45071 every 8 connection(s)
I20260927 04:03:16.072364 23568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:16.073415 23635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:16.075567 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 23568
I20260927 04:03:16.075681 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:16.075907 18147 external_mini_cluster.cc:1428] Setting key 69f03b8eb3216c40d8a516639f74b9a6
I20260927 04:03:16.076331 23635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392: Bootstrap starting.
I20260927 04:03:16.076989 23635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:16.077317 23635 log.cc:824] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:16.078239 23635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392: No bootstrap required, opened a new log
I20260927 04:03:16.080874 23635 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2e34831c424b4788784017dea19392" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45071 } }
I20260927 04:03:16.081075 23635 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:16.081123 23635 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2e34831c424b4788784017dea19392, State: Initialized, Role: FOLLOWER
I20260927 04:03:16.081331 23635 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2e34831c424b4788784017dea19392" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45071 } }
I20260927 04:03:16.081449 23635 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:16.081487 23635 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:16.081532 23635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:16.082231 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:16.076506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41242 (local address 127.17.184.254:45071)
0927 04:03:16.076697 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:16.076704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:16.076737 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:03:16.077218 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:16.077225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:16.077286 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:16.077399 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:16.077799 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:16.078797 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:16.080085 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:16.080499 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:16.080593 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:16.081461 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:16.081465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:16.081466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:16.081694 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:16.081752 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:16.081768 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:16.081845 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:16.081922 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 04:03:16.082006 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260927 04:03:16.082350 23635 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2e34831c424b4788784017dea19392" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45071 } }
I20260927 04:03:16.082448 23635 leader_election.cc:302] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec2e34831c424b4788784017dea19392; no voters:
I20260927 04:03:16.082643 23635 leader_election.cc:288] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:16.082674 23640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:16.082863 23640 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [term 1 LEADER]: Becoming Leader. State: Replica: ec2e34831c424b4788784017dea19392, State: Running, Role: LEADER
I20260927 04:03:16.082932 23635 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:16.082974 23640 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2e34831c424b4788784017dea19392" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45071 } }
I20260927 04:03:16.083315 23641 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec2e34831c424b4788784017dea19392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2e34831c424b4788784017dea19392" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45071 } } }
I20260927 04:03:16.083346 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:03:16.083388 23641 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:16.083587 23642 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec2e34831c424b4788784017dea19392. Latest consensus state: current_term: 1 leader_uuid: "ec2e34831c424b4788784017dea19392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec2e34831c424b4788784017dea19392" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45071 } } }
I20260927 04:03:16.083986 23642 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:16.084131 23645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:16.084591 23645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:16.086387 23645 catalog_manager.cc:1347] Generated new cluster ID: 5aeeb4fbcdd445419c030158bfaffb5c
I20260927 04:03:16.086464 23645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:16.100611 23645 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:16.101423 23645 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:16.111019 23645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec2e34831c424b4788784017dea19392: Generated new TSK 0
I20260927 04:03:16.111272 23645 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:03:16.173877 23638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:16.120245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.17.184.254:45071)
0927 04:03:16.120308 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:16.120310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:16.120412 (+ 102us) server_negotiation.cc:407] Connection header received
0927 04:03:16.120472 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:16.120473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:16.120486 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:16.120568 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0927 04:03:16.120693 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:16.121473 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:16.122358 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:16.122673 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:16.122747 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:16.122936 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 04:03:16.173589 (+ 50653us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 04:03:16.173794 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41252: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 04:03:16.173960 23638 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41252: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 04:03:16.174034 23664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:16.120151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:45071 (local address 127.0.0.1:41252)
0927 04:03:16.120343 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:16.120354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:16.120412 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:16.120573 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:16.120574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:16.120576 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0927 04:03:16.120630 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:16.120633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:16.121561 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:16.121562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:16.122268 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:16.122271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:16.122759 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:16.122760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:16.122808 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:16.122813 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 04:03:16.173776 (+ 50963us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 04:03:16.173902 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.184.254:45071: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":117,"threads_started":1}
I20260927 04:03:16.174603 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 23568
2026-09-27T04:03:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (238 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:16 dist-test-slave-zqrn krb5kdc[23676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:16 dist-test-slave-zqrn krb5kdc[23676](info): set up 2 sockets
Sep 27 04:03:16 dist-test-slave-zqrn krb5kdc[23676](info): commencing operation
krb5kdc: starting...
W20260927 04:03:17.451324 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:17 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:17.577834 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33763
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:33763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:17.665104 23689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:17.665401 23689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:17.665442 23689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:17.666713 23689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:17.666790 23689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:17.666822 23689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:17.666857 23689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:17.666885 23689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:17.668504 23689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:33763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:17.668812 23689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:17.669068 23689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:17.674340 23694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:17.674324 23697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:17.674324 23695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:17.674638 23689 server_base.cc:1034] running on GCE node
I20260927 04:03:17.675034 23689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:17.675366 23689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:17.676537 23689 hybrid_clock.cc:648] HybridClock initialized: now 1790481797676468 us; error 90 us; skew 500 ppm
Sep 27 04:03:17 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:17.679557 23689 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:17.680073 23689 webserver.cc:460] Webserver started at http://127.17.184.254:43553/ using document root <none> and password file <none>
I20260927 04:03:17.680303 23689 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:17.680348 23689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:17.680440 23689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:17.681519 23689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "74cf4cdb9a5b40a3abed4bcfd312b115"
format_stamp: "Formatted at 2026-09-27 04:03:17 on dist-test-slave-zqrn"
server_key: "72b741d7822e101e84e65835e0913a68"
server_key_iv: "f0bbc3a04e2ebfabcef416b658fa9faa"
server_key_version: "encryptionkey@0"
I20260927 04:03:17.681839 23689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "74cf4cdb9a5b40a3abed4bcfd312b115"
format_stamp: "Formatted at 2026-09-27 04:03:17 on dist-test-slave-zqrn"
server_key: "72b741d7822e101e84e65835e0913a68"
server_key_iv: "f0bbc3a04e2ebfabcef416b658fa9faa"
server_key_version: "encryptionkey@0"
I20260927 04:03:17.683236 23689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:03:17.684266 23704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:17.684520 23689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:17.684635 23689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "74cf4cdb9a5b40a3abed4bcfd312b115"
format_stamp: "Formatted at 2026-09-27 04:03:17 on dist-test-slave-zqrn"
server_key: "72b741d7822e101e84e65835e0913a68"
server_key_iv: "f0bbc3a04e2ebfabcef416b658fa9faa"
server_key_version: "encryptionkey@0"
I20260927 04:03:17.684706 23689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:17.697305 23689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:17.699805 23689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:17.699960 23689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:17.704124 23689 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:33763
I20260927 04:03:17.704159 23756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:33763 every 8 connection(s)
I20260927 04:03:17.704525 23689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:17.705477 23757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:17.707955 23757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115: Bootstrap starting.
I20260927 04:03:17.708556 23757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:17.708858 23757 log.cc:824] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:17.709587 23757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115: No bootstrap required, opened a new log
I20260927 04:03:17.711342 23757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cf4cdb9a5b40a3abed4bcfd312b115" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33763 } }
I20260927 04:03:17.711511 23757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:17.711560 23757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cf4cdb9a5b40a3abed4bcfd312b115, State: Initialized, Role: FOLLOWER
I20260927 04:03:17.711736 23757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cf4cdb9a5b40a3abed4bcfd312b115" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33763 } }
I20260927 04:03:17.711846 23757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:17.711891 23757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:17.711941 23757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:17.712517 23757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cf4cdb9a5b40a3abed4bcfd312b115" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33763 } }
I20260927 04:03:17.712641 23757 leader_election.cc:302] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74cf4cdb9a5b40a3abed4bcfd312b115; no voters:
I20260927 04:03:17.712842 23757 leader_election.cc:288] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:17.712965 23760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:17.713243 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 23689
I20260927 04:03:17.713348 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:17.713351 23760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [term 1 LEADER]: Becoming Leader. State: Replica: 74cf4cdb9a5b40a3abed4bcfd312b115, State: Running, Role: LEADER
I20260927 04:03:17.713438 23760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cf4cdb9a5b40a3abed4bcfd312b115" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33763 } }
I20260927 04:03:17.713542 18147 external_mini_cluster.cc:1428] Setting key 589d6bfda8043a34aecc721fcabb1042
I20260927 04:03:17.713683 23757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:17.713886 23761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74cf4cdb9a5b40a3abed4bcfd312b115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cf4cdb9a5b40a3abed4bcfd312b115" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33763 } } }
I20260927 04:03:17.714133 23761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:17.714066 23762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74cf4cdb9a5b40a3abed4bcfd312b115. Latest consensus state: current_term: 1 leader_uuid: "74cf4cdb9a5b40a3abed4bcfd312b115" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cf4cdb9a5b40a3abed4bcfd312b115" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33763 } } }
I20260927 04:03:17.714352 23762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:17.714515 23768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:17.715219 23768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:17.717044 23768 catalog_manager.cc:1347] Generated new cluster ID: ffda25f772e84d858adb55626e29bc06
I20260927 04:03:17.717097 23768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:03:17 dist-test-slave-zqrn krb5kdc[23676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:17.724824 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:17.714197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.17.184.254:33763)
0927 04:03:17.714936 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:17.714948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:17.714988 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:03:17.715135 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:17.715144 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:17.715211 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:17.716162 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:17.716260 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:17.717469 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:17.718285 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:17.718746 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:17.718865 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:17.721288 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:17.721292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:17.721292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:17.721314 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:17.723190 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:17.723577 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:17.723581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:17.723581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:17.723694 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:17.723904 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:17.723905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:17.723906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:17.724193 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:17.724374 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:17.724449 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 04:03:17.724596 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":87,"threads_started":1}
I20260927 04:03:17.732949 23768 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:17.734077 23768 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:17.738755 23768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74cf4cdb9a5b40a3abed4bcfd312b115: Generated new TSK 0
I20260927 04:03:17.739003 23768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:03:17.776117 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33763
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:17.865289 23783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:17.865532 23783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:17.865581 23783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:17.866951 23783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:17.867012 23783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:17.867043 23783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:17.868654 23783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:17.868932 23783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:17.869192 23783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:17.874728 23788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:17.874756 23789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:17.874980 23783 server_base.cc:1034] running on GCE node
W20260927 04:03:17.874751 23791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:17.875238 23783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:17.875514 23783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:17.876690 23783 hybrid_clock.cc:648] HybridClock initialized: now 1790481797876680 us; error 33 us; skew 500 ppm
Sep 27 04:03:17 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:17.879489 23783 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:03:17.879964 23783 webserver.cc:460] Webserver started at http://127.17.184.193:41879/ using document root <none> and password file <none>
I20260927 04:03:17.880199 23783 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:17.880275 23783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:17.880460 23783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:17.881647 23783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "419a23d13deb4a1ca42418401ea90993"
format_stamp: "Formatted at 2026-09-27 04:03:17 on dist-test-slave-zqrn"
server_key: "a8eec425c3b778fafeb3d1416f1d3d28"
server_key_iv: "e8ac7e25efc4f53bcb17122be7084a38"
server_key_version: "encryptionkey@0"
I20260927 04:03:17.882117 23783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "419a23d13deb4a1ca42418401ea90993"
format_stamp: "Formatted at 2026-09-27 04:03:17 on dist-test-slave-zqrn"
server_key: "a8eec425c3b778fafeb3d1416f1d3d28"
server_key_iv: "e8ac7e25efc4f53bcb17122be7084a38"
server_key_version: "encryptionkey@0"
I20260927 04:03:17.883621 23783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:17.884495 23798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:17.884725 23783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:17.884835 23783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "419a23d13deb4a1ca42418401ea90993"
format_stamp: "Formatted at 2026-09-27 04:03:17 on dist-test-slave-zqrn"
server_key: "a8eec425c3b778fafeb3d1416f1d3d28"
server_key_iv: "e8ac7e25efc4f53bcb17122be7084a38"
server_key_version: "encryptionkey@0"
I20260927 04:03:17.884915 23783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:17.899582 23783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:17.902122 23783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:17.902346 23783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:17.902623 23783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:17.903017 23783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:17.903075 23783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:17.903121 23783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:17.903148 23783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:17.909364 23783 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:42309
I20260927 04:03:17.909449 23911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:42309 every 8 connection(s)
I20260927 04:03:17.909821 23783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:17.911816 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 23783
I20260927 04:03:17.911936 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:17.912134 18147 external_mini_cluster.cc:1428] Setting key 82c4ee0fe99d52d0d499fb6b45371702
Sep 27 04:03:17 dist-test-slave-zqrn krb5kdc[23676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481797, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:03:17.922693 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:17.910680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:48669 (local address 127.17.184.254:33763)
0927 04:03:17.910757 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:17.910762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:17.910955 (+ 193us) server_negotiation.cc:407] Connection header received
0927 04:03:17.912205 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:17.912208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:17.912224 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:17.912280 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:17.913370 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:17.914069 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:17.915670 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:17.916059 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:17.916148 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:17.918741 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:17.918743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:17.918744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:17.918759 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:17.920833 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:17.921360 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:17.921363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:17.921363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:17.921440 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:17.921780 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:17.921782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:17.921782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:17.921973 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:17.922055 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:17.922442 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 04:03:17.922558 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:03:17.922737 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:17.910634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33763 (local address 127.17.184.193:48669)
0927 04:03:17.910801 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:17.910846 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:17.912042 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:17.912412 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:17.912417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:17.912720 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:17.913222 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:17.913231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:17.914171 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:17.914172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:17.915573 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:17.915576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:17.916176 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:17.916178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:17.916313 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:17.916966 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:17.916976 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:17.918622 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:17.920982 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:17.920990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:17.920992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:17.921241 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:17.921558 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:17.921559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:17.921560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:17.921692 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:17.922124 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:17.922130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:17.922288 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 04:03:17.922390 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 04:03:17.922539 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":105,"threads_started":1}
I20260927 04:03:17.923564 23912 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33763
I20260927 04:03:17.923679 23912 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:17.923866 23912 heartbeater.cc:511] Master 127.17.184.254:33763 requested a full tablet report, sending...
I20260927 04:03:17.924445 23721 ts_manager.cc:194] Registered new tserver with Master: 419a23d13deb4a1ca42418401ea90993 (127.17.184.193:42309)
I20260927 04:03:17.925149 23721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:48669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:03:17.944056 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33763
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:18.033777 23917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:18.034070 23917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:18.034113 23917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:18.035734 23917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:18.035835 23917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:18.035887 23917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:18.037851 23917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:18.038213 23917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:18.038486 23917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:18.044229 23922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:18.044229 23923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:18.044437 23917 server_base.cc:1034] running on GCE node
W20260927 04:03:18.044229 23925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:18.044781 23917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:18.045058 23917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:18.046295 23917 hybrid_clock.cc:648] HybridClock initialized: now 1790481798046264 us; error 43 us; skew 500 ppm
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.049453 23917 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:03:18.050046 23917 webserver.cc:460] Webserver started at http://127.17.184.194:43623/ using document root <none> and password file <none>
I20260927 04:03:18.050396 23917 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:18.050465 23917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:18.050634 23917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:18.052193 23917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "651bc1c3674d46b88a504f5fa893c27b"
format_stamp: "Formatted at 2026-09-27 04:03:18 on dist-test-slave-zqrn"
server_key: "f366f294a477c4549eedcb8234a61735"
server_key_iv: "0263780f9c327bb82f15e16040e09dd5"
server_key_version: "encryptionkey@0"
I20260927 04:03:18.052809 23917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "651bc1c3674d46b88a504f5fa893c27b"
format_stamp: "Formatted at 2026-09-27 04:03:18 on dist-test-slave-zqrn"
server_key: "f366f294a477c4549eedcb8234a61735"
server_key_iv: "0263780f9c327bb82f15e16040e09dd5"
server_key_version: "encryptionkey@0"
I20260927 04:03:18.054595 23917 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 04:03:18.055688 23932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:18.056186 23917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:18.056272 23917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "651bc1c3674d46b88a504f5fa893c27b"
format_stamp: "Formatted at 2026-09-27 04:03:18 on dist-test-slave-zqrn"
server_key: "f366f294a477c4549eedcb8234a61735"
server_key_iv: "0263780f9c327bb82f15e16040e09dd5"
server_key_version: "encryptionkey@0"
I20260927 04:03:18.056391 23917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:18.085076 23917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:18.087833 23917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:18.087960 23917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:18.088330 23917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:18.088706 23917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:18.088765 23917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:18.088897 23917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:18.088955 23917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:18.095801 23917 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:46005
I20260927 04:03:18.095954 24045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:46005 every 8 connection(s)
I20260927 04:03:18.096221 23917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:18.100507 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 23917
I20260927 04:03:18.100623 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:18.100801 18147 external_mini_cluster.cc:1428] Setting key d94cd8be8e5dee7eb4c7e1a81e8c3d1f
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:18.109462 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:18.097992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:55199 (local address 127.17.184.254:33763)
0927 04:03:18.098044 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:18.098046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:18.098281 (+ 235us) server_negotiation.cc:407] Connection header received
0927 04:03:18.099274 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:18.099276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:18.099289 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:18.099356 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:18.100088 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.100919 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.102259 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.102582 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.102664 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:18.105590 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:18.105592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:18.105593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:18.105603 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:18.107802 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.108258 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.108259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.108260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.108332 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.108639 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.108640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.108641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.108803 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:18.108880 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:18.109210 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 04:03:18.109328 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:03:18.109560 24048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:18.097932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33763 (local address 127.17.184.194:55199)
0927 04:03:18.098107 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:18.098152 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:18.099176 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:18.099439 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:18.099443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:18.099723 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:18.099981 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:18.099985 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.101043 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.101045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:18.102162 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:18.102165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.102787 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.102788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:18.103025 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:18.103635 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:18.103644 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:18.105481 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:18.107907 (+ 2426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:18.107916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:18.107918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:18.108167 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:18.108430 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:18.108431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:18.108432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:18.108560 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:18.108901 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:18.108907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:18.109046 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 04:03:18.109167 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 04:03:18.109329 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
I20260927 04:03:18.110121 24046 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33763
I20260927 04:03:18.110258 24046 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:18.110453 24046 heartbeater.cc:511] Master 127.17.184.254:33763 requested a full tablet report, sending...
I20260927 04:03:18.110956 23721 ts_manager.cc:194] Registered new tserver with Master: 651bc1c3674d46b88a504f5fa893c27b (127.17.184.194:46005)
I20260927 04:03:18.111534 23721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:55199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:03:18.131516 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33763
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:18.213125 24051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:18.213368 24051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:18.213395 24051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:18.214927 24051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:18.214988 24051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:18.215018 24051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:18.216666 24051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:18.216929 24051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:18.217168 24051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:18.222512 24056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:18.222589 24051 server_base.cc:1034] running on GCE node
W20260927 04:03:18.222512 24057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:18.222770 24059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:18.223034 24051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:18.223299 24051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:18.224454 24051 hybrid_clock.cc:648] HybridClock initialized: now 1790481798224431 us; error 41 us; skew 500 ppm
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.227644 24051 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:03:18.228122 24051 webserver.cc:460] Webserver started at http://127.17.184.195:43359/ using document root <none> and password file <none>
I20260927 04:03:18.228359 24051 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:18.228405 24051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:18.228574 24051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:18.229641 24051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "ffc30d4ff8ba405390c771f81d742781"
format_stamp: "Formatted at 2026-09-27 04:03:18 on dist-test-slave-zqrn"
server_key: "94262da1e9119a3999e4d02aae83428d"
server_key_iv: "ec870a64756489442bd101c163820c5f"
server_key_version: "encryptionkey@0"
I20260927 04:03:18.230003 24051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "ffc30d4ff8ba405390c771f81d742781"
format_stamp: "Formatted at 2026-09-27 04:03:18 on dist-test-slave-zqrn"
server_key: "94262da1e9119a3999e4d02aae83428d"
server_key_iv: "ec870a64756489442bd101c163820c5f"
server_key_version: "encryptionkey@0"
I20260927 04:03:18.231421 24051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:18.232218 24066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:18.232419 24051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:18.232529 24051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "ffc30d4ff8ba405390c771f81d742781"
format_stamp: "Formatted at 2026-09-27 04:03:18 on dist-test-slave-zqrn"
server_key: "94262da1e9119a3999e4d02aae83428d"
server_key_iv: "ec870a64756489442bd101c163820c5f"
server_key_version: "encryptionkey@0"
I20260927 04:03:18.232604 24051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:18.248737 24051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:18.250859 24051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:18.251044 24051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:18.251303 24051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:18.251624 24051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:18.251679 24051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:18.251725 24051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:18.251756 24051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:18.257970 24051 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:46531
I20260927 04:03:18.258024 24179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:46531 every 8 connection(s)
I20260927 04:03:18.258383 24051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:18.266734 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 24051
I20260927 04:03:18.266846 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:18.267059 18147 external_mini_cluster.cc:1428] Setting key be0c078bc33bb013b3cefa0084a968a7
I20260927 04:03:18.268066 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:18.259185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:54275 (local address 127.17.184.254:33763)
0927 04:03:18.259267 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:18.259270 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:18.259462 (+ 192us) server_negotiation.cc:407] Connection header received
0927 04:03:18.260456 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:18.260457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:18.260468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:18.260520 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:18.261119 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.261614 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.262565 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.262774 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.262872 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:18.265078 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:18.265080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:18.265080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:18.265087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:18.266394 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.266924 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.266926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.266926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.266989 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.267316 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.267318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.267318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.267481 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:18.267555 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:18.267889 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 04:03:18.267977 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 04:03:18.268313 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:18.259146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33763 (local address 127.17.184.195:54275)
0927 04:03:18.259340 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:18.259378 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:18.260365 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:18.260575 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:18.260578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:18.260838 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:18.261035 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:18.261039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.261695 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.261696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:18.262483 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:18.262484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.263015 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.263016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:18.263127 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:18.263622 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:18.263630 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:18.264992 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:18.266491 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:18.266498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:18.266500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:18.266802 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:18.267067 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:18.267068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:18.267068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:18.267238 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:18.267552 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:18.267558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:18.267729 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 04:03:18.267852 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 04:03:18.268014 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260927 04:03:18.268729 24180 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33763
I20260927 04:03:18.268827 24180 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:18.268968 24180 heartbeater.cc:511] Master 127.17.184.254:33763 requested a full tablet report, sending...
I20260927 04:03:18.269378 23721 ts_manager.cc:194] Registered new tserver with Master: ffc30d4ff8ba405390c771f81d742781 (127.17.184.195:46531)
I20260927 04:03:18.269704 23721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:54275
I20260927 04:03:18.279129 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/test-admin.keytab.
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.296891 18147 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 04:03:18.298785 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:18.297897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.17.184.193:42309)
0927 04:03:18.298103 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:18.298108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:18.298119 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:18.298145 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:18.298147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:18.298162 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:18.298231 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:18.298686 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48304: BlockingRecv error: recv got EOF from 127.0.0.1:48304 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":119,"threads_started":1}
W20260927 04:03:18.298854 18147 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5cc' of type FILE
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.299935 18147 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.300889 18147 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:03:18.309764 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:18.301534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.17.184.193:42309)
0927 04:03:18.301590 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:18.301593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:18.301660 (+ 67us) server_negotiation.cc:407] Connection header received
0927 04:03:18.301756 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:18.301757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:18.301774 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:18.301885 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:18.302322 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.303120 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.303902 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.304240 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.304326 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:18.306553 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:18.306564 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:18.306565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:18.306577 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:18.308467 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.308885 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.308896 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.308896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.308950 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.309151 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.309152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.309152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.309301 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:18.309452 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:18.309535 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 04:03:18.309621 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 04:03:18.311791 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:18.311112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48332 (local address 127.17.184.193:42309)
0927 04:03:18.311170 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:18.311172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:18.311255 (+ 83us) server_negotiation.cc:407] Connection header received
0927 04:03:18.311334 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:18.311335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:18.311345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:18.311387 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:18.311681 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48332: BlockingRecv error: recv got EOF from 127.0.0.1:48332 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 04:03:18.311851 18147 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5cc' of type FILE
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.312891 18147 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.313776 18147 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:03:18.321100 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:18.314375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.17.184.193:42309)
0927 04:03:18.314429 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:18.314432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:18.314493 (+ 61us) server_negotiation.cc:407] Connection header received
0927 04:03:18.314560 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:18.314561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:18.314569 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:18.314638 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:18.315086 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.315584 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.316313 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.316599 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.316678 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:18.318719 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:18.318721 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:18.318721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:18.318728 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:18.319983 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.320316 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.320317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.320318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.320375 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.320573 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.320574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.320574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.320681 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:18.320809 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:18.320911 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 04:03:18.320993 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 04:03:18.322752 24187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:18.322206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.17.184.193:42309)
0927 04:03:18.322259 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:18.322262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:18.322309 (+ 47us) server_negotiation.cc:407] Connection header received
0927 04:03:18.322378 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:18.322379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:18.322389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:18.322435 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:18.322696 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48352: BlockingRecv error: recv got EOF from 127.0.0.1:48352 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 04:03:18.322837 18147 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790481773036938-18147-0/krb5kdc/krb5cc' of type FILE
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.323998 18147 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:18.324913 18147 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[23676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:03:18.332384 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:18.325463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.17.184.193:42309)
0927 04:03:18.325520 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:18.325521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:18.325578 (+ 57us) server_negotiation.cc:407] Connection header received
0927 04:03:18.325650 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:18.325651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:18.325660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:18.325774 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:18.326145 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.326609 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.327390 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:18.327579 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:18.327681 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:18.329728 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:18.329730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:18.329730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:18.329737 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:18.331264 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.331562 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.331563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.331563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.331613 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:18.331775 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:18.331787 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:18.331787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:18.331878 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:18.331957 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:18.332132 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 04:03:18.332250 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 04:03:18.333036 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 23783
I20260927 04:03:18.338337 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 23917
I20260927 04:03:18.342839 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 24051
I20260927 04:03:18.348824 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 23689
2026-09-27T04:03:18Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1996 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[24200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[24200](info): set up 2 sockets
Sep 27 04:03:18 dist-test-slave-zqrn krb5kdc[24200](info): commencing operation
krb5kdc: starting...
W20260927 04:03:19.415720 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:19 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.254.keytab.
Entry for principal oryx/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.254.keytab.
I20260927 04:03:19.542006 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:34757
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.254.keytab
--principal=oryx/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:34757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:19.629357 24213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:19.629570 24213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:19.629594 24213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:19.630707 24213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:19.630741 24213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:19.630755 24213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:19.630769 24213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:19.630780 24213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:19.632300 24213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:34757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.254.keytab
--principal=oryx/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:34757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:19.632525 24213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:19.632699 24213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:19.637571 24221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:19.637595 24219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:19.637571 24218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:19.638163 24213 server_base.cc:1034] running on GCE node
I20260927 04:03:19.638437 24213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:19.638682 24213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:19.639823 24213 hybrid_clock.cc:648] HybridClock initialized: now 1790481799639805 us; error 32 us; skew 500 ppm
Sep 27 04:03:19 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:19.647115 24213 init.cc:377] Logged in from keytab as oryx/127.17.184.254@KRBTEST.COM (short username oryx)
I20260927 04:03:19.647620 24213 webserver.cc:460] Webserver started at http://127.17.184.254:43247/ using document root <none> and password file <none>
I20260927 04:03:19.647869 24213 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:19.647915 24213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:19.648013 24213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:19.649061 24213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "4ba21c2b6e0047109248e6ca90d28c85"
format_stamp: "Formatted at 2026-09-27 04:03:19 on dist-test-slave-zqrn"
server_key: "58fe33be16706fce96e5b8e405a6b44e"
server_key_iv: "66770440a6bf774d9581508a5d590d83"
server_key_version: "encryptionkey@0"
I20260927 04:03:19.649365 24213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "4ba21c2b6e0047109248e6ca90d28c85"
format_stamp: "Formatted at 2026-09-27 04:03:19 on dist-test-slave-zqrn"
server_key: "58fe33be16706fce96e5b8e405a6b44e"
server_key_iv: "66770440a6bf774d9581508a5d590d83"
server_key_version: "encryptionkey@0"
I20260927 04:03:19.650676 24213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:19.651624 24228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:19.651829 24213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:19.651926 24213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "4ba21c2b6e0047109248e6ca90d28c85"
format_stamp: "Formatted at 2026-09-27 04:03:19 on dist-test-slave-zqrn"
server_key: "58fe33be16706fce96e5b8e405a6b44e"
server_key_iv: "66770440a6bf774d9581508a5d590d83"
server_key_version: "encryptionkey@0"
I20260927 04:03:19.652001 24213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:19.688544 24213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:19.691093 24213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:19.691211 24213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:19.694986 24213 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:34757
I20260927 04:03:19.695037 24280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:34757 every 8 connection(s)
I20260927 04:03:19.695389 24213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:19.696409 24281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:19.697489 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 24213
I20260927 04:03:19.697604 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:19.697808 18147 external_mini_cluster.cc:1428] Setting key 72d419943c5a45e4bccf92ce2f8c9e64
I20260927 04:03:19.699400 24281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: Bootstrap starting.
I20260927 04:03:19.700016 24281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:19.700318 24281 log.cc:824] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:19.701095 24281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: No bootstrap required, opened a new log
I20260927 04:03:19.703001 24281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } }
I20260927 04:03:19.703164 24281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:19.703207 24281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba21c2b6e0047109248e6ca90d28c85, State: Initialized, Role: FOLLOWER
I20260927 04:03:19.703332 24281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } }
I20260927 04:03:19.703446 24281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:19.703491 24281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:19.703543 24281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:19.704115 24281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } }
I20260927 04:03:19.704241 24281 leader_election.cc:302] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ba21c2b6e0047109248e6ca90d28c85; no voters:
I20260927 04:03:19.704413 24281 leader_election.cc:288] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:19.704452 24286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:19.704614 24286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 1 LEADER]: Becoming Leader. State: Replica: 4ba21c2b6e0047109248e6ca90d28c85, State: Running, Role: LEADER
I20260927 04:03:19.704696 24281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:19.704713 24286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } }
Sep 27 04:03:19 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:19.705075 24286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ba21c2b6e0047109248e6ca90d28c85. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } }
I20260927 04:03:19.705080 24288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } }
I20260927 04:03:19.705170 24286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:19.705189 24288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:19.706290 24292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:19.707794 24292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:19.709293 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:19.698422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.17.184.254:34757)
0927 04:03:19.698631 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:19.698645 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:19.698715 (+ 70us) server_negotiation.cc:407] Connection header received
0927 04:03:19.699229 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:19.699237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:19.699290 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:19.699439 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:19.700082 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:19.701315 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:19.702104 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:19.702525 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:19.702650 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:19.705492 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:19.705497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:19.705497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:19.705510 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:19.707714 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:19.708102 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:19.708106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:19.708106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:19.708200 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:19.708413 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:19.708414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:19.708414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:19.708656 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:19.708829 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:19.708909 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 04:03:19.709055 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260927 04:03:19.709619 24292 catalog_manager.cc:1347] Generated new cluster ID: c86a0064840946bcb8825eaa90b1706f
I20260927 04:03:19.709673 24292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:19.722101 24292 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:19.722654 24292 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:19.726577 24292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: Generated new TSK 0
I20260927 04:03:19.726778 24292 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab.
Entry for principal oryx/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab.
I20260927 04:03:19.767694 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:34757
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab
--principal=oryx/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:19.850608 24307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:19.851083 24307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:19.851332 24307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:19.852775 24307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:19.852838 24307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:19.852869 24307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:19.854439 24307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab
--principal=oryx/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:34757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:19.854698 24307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:19.855037 24307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:19.860282 24313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:19.860282 24312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:19.860457 24315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:19.860600 24307 server_base.cc:1034] running on GCE node
I20260927 04:03:19.860877 24307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:19.861137 24307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:19.862324 24307 hybrid_clock.cc:648] HybridClock initialized: now 1790481799862301 us; error 61 us; skew 500 ppm
Sep 27 04:03:19 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:19.865190 24307 init.cc:377] Logged in from keytab as oryx/127.17.184.193@KRBTEST.COM (short username oryx)
I20260927 04:03:19.865659 24307 webserver.cc:460] Webserver started at http://127.17.184.193:41321/ using document root <none> and password file <none>
I20260927 04:03:19.865890 24307 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:19.865960 24307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:19.866135 24307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:19.867316 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "60d807011b38409097e30e028fb20cbb"
format_stamp: "Formatted at 2026-09-27 04:03:19 on dist-test-slave-zqrn"
server_key: "36ada9b134d63c3d1789153e1eb6f67a"
server_key_iv: "3dc3af73021e6a6b8bc5e90622242cd1"
server_key_version: "encryptionkey@0"
I20260927 04:03:19.867683 24307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "60d807011b38409097e30e028fb20cbb"
format_stamp: "Formatted at 2026-09-27 04:03:19 on dist-test-slave-zqrn"
server_key: "36ada9b134d63c3d1789153e1eb6f67a"
server_key_iv: "3dc3af73021e6a6b8bc5e90622242cd1"
server_key_version: "encryptionkey@0"
I20260927 04:03:19.869282 24307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 04:03:19.870101 24322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:19.870353 24307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:19.870464 24307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "60d807011b38409097e30e028fb20cbb"
format_stamp: "Formatted at 2026-09-27 04:03:19 on dist-test-slave-zqrn"
server_key: "36ada9b134d63c3d1789153e1eb6f67a"
server_key_iv: "3dc3af73021e6a6b8bc5e90622242cd1"
server_key_version: "encryptionkey@0"
I20260927 04:03:19.870543 24307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:19.889794 24307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:19.892349 24307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:19.892547 24307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:19.892846 24307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:19.893198 24307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:19.893236 24307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:19.893261 24307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:19.893297 24307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:19.899827 24307 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:33249
I20260927 04:03:19.899868 24435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:33249 every 8 connection(s)
I20260927 04:03:19.900252 24307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:19.902832 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 24307
I20260927 04:03:19.903002 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:19.903273 18147 external_mini_cluster.cc:1428] Setting key 1c87839b1efc16173da33f14349cdc50
Sep 27 04:03:19 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:03:19.912017 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:19.901190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:41687 (local address 127.17.184.254:34757)
0927 04:03:19.901237 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:19.901239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:19.901491 (+ 252us) server_negotiation.cc:407] Connection header received
0927 04:03:19.902075 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:19.902076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:19.902088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:19.902153 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:19.903466 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:19.904198 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:19.905695 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:19.906001 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:19.906084 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:19.908799 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:19.908801 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:19.908801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:19.908810 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:19.910369 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:19.910888 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:19.910889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:19.910912 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:19.910981 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:19.911242 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:19.911243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:19.911244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:19.911377 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:19.911457 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:19.911788 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 04:03:19.911920 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:03:19.912184 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:19.901150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:41687)
0927 04:03:19.901342 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:19.901388 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:19.901955 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:19.902231 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:19.902235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:19.902988 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:19.903271 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:19.903278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:19.904367 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:19.904368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:19.905528 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:19.905531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:19.906088 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:19.906089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:19.906248 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:19.906830 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:19.906838 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:19.908658 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:19.910497 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:19.910504 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:19.910507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:19.910761 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:19.911055 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:19.911056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:19.911057 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:19.911170 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:19.911447 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:19.911454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:19.911632 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 04:03:19.911766 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 04:03:19.911934 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":112,"threads_started":1}
I20260927 04:03:19.912895 24436 heartbeater.cc:347] Connected to a master server at 127.17.184.254:34757
I20260927 04:03:19.913024 24436 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:19.913231 24436 heartbeater.cc:511] Master 127.17.184.254:34757 requested a full tablet report, sending...
I20260927 04:03:19.913839 24245 ts_manager.cc:194] Registered new tserver with Master: 60d807011b38409097e30e028fb20cbb (127.17.184.193:33249)
I20260927 04:03:19.914439 24245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:41687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab.
Entry for principal oryx/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab.
I20260927 04:03:19.931694 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:34757
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab
--principal=oryx/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:20.017586 24441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:20.017925 24441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:20.017971 24441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:20.019508 24441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:20.019590 24441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:20.019640 24441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:20.022142 24441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab
--principal=oryx/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:34757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:20.022640 24441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:20.022884 24441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:20.028491 24449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.028493 24447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.028537 24446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:20.028698 24441 server_base.cc:1034] running on GCE node
I20260927 04:03:20.029029 24441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:20.029291 24441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:20.030457 24441 hybrid_clock.cc:648] HybridClock initialized: now 1790481800030455 us; error 41 us; skew 500 ppm
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:20.033247 24441 init.cc:377] Logged in from keytab as oryx/127.17.184.194@KRBTEST.COM (short username oryx)
I20260927 04:03:20.033728 24441 webserver.cc:460] Webserver started at http://127.17.184.194:45577/ using document root <none> and password file <none>
I20260927 04:03:20.033988 24441 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.034088 24441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.034232 24441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:20.035424 24441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "e84555cae7414ffc947a877ac4315649"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "0a79483e90e67f5a2e8c30f486e80d6d"
server_key_iv: "cdfd8b7acd86bc6d69b1afef39233928"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.035789 24441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "e84555cae7414ffc947a877ac4315649"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "0a79483e90e67f5a2e8c30f486e80d6d"
server_key_iv: "cdfd8b7acd86bc6d69b1afef39233928"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.037269 24441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:20.038275 24456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.038499 24441 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:20.038609 24441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "e84555cae7414ffc947a877ac4315649"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "0a79483e90e67f5a2e8c30f486e80d6d"
server_key_iv: "cdfd8b7acd86bc6d69b1afef39233928"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.038690 24441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.059262 24441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:20.061676 24441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:20.061853 24441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:20.062114 24441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:20.062434 24441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:20.062498 24441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.062544 24441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:20.062566 24441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.068740 24441 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:45969
I20260927 04:03:20.068892 24569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:45969 every 8 connection(s)
I20260927 04:03:20.069164 24441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:20.077028 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 24441
I20260927 04:03:20.077144 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:20.077369 18147 external_mini_cluster.cc:1428] Setting key 20536214bacc557004a61adeacc22747
I20260927 04:03:20.079569 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.069983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:35765 (local address 127.17.184.254:34757)
0927 04:03:20.070066 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.070069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.070266 (+ 197us) server_negotiation.cc:407] Connection header received
0927 04:03:20.070858 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.070859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.070871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.071009 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.071977 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.072506 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.073669 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.073866 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.073936 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.076370 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.076371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.076371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.076378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.078113 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.078526 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.078528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.078528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.078594 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.078853 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.078854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.078854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.079013 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.079091 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.079378 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.079473 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 04:03:20.079811 24572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.069944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:35765)
0927 04:03:20.070136 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.070175 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.070737 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.071073 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.071076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.071664 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.071875 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.071880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.072601 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.072602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.073551 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.073554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.073955 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.073956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.074061 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.074718 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.074727 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.076261 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.078179 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.078188 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.078191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.078438 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.078664 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.078665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.078665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.078783 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.079082 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.079089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.079238 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.079355 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.079524 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":113,"threads_started":1}
I20260927 04:03:20.080349 24570 heartbeater.cc:347] Connected to a master server at 127.17.184.254:34757
I20260927 04:03:20.080470 24570 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:20.080683 24570 heartbeater.cc:511] Master 127.17.184.254:34757 requested a full tablet report, sending...
I20260927 04:03:20.081043 24245 ts_manager.cc:194] Registered new tserver with Master: e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969)
I20260927 04:03:20.081439 24245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:35765
WARNING: no policy specified for oryx/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab.
Entry for principal oryx/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab.
I20260927 04:03:20.106859 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:34757
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab
--principal=oryx/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:20.190842 24575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:20.191076 24575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:20.191102 24575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:20.192299 24575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:20.192360 24575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:20.192387 24575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:20.193827 24575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab
--principal=oryx/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:34757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:20.194022 24575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:20.194202 24575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:20.199463 24580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.199496 24581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.199460 24583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:20.199803 24575 server_base.cc:1034] running on GCE node
I20260927 04:03:20.200058 24575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:20.200342 24575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:20.201517 24575 hybrid_clock.cc:648] HybridClock initialized: now 1790481800201511 us; error 38 us; skew 500 ppm
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:20.204241 24575 init.cc:377] Logged in from keytab as oryx/127.17.184.195@KRBTEST.COM (short username oryx)
I20260927 04:03:20.204795 24575 webserver.cc:460] Webserver started at http://127.17.184.195:40753/ using document root <none> and password file <none>
I20260927 04:03:20.205044 24575 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.205118 24575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.205265 24575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:20.206449 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "5d155427f4504ceca4f5a00791005a3b"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "ac05361d97b99d90c105b6f791507762"
server_key_iv: "be193c294ae3d7a091b3099fd0c5989c"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.206854 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "5d155427f4504ceca4f5a00791005a3b"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "ac05361d97b99d90c105b6f791507762"
server_key_iv: "be193c294ae3d7a091b3099fd0c5989c"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.208232 24575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:20.209017 24590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.209256 24575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:20.209372 24575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "5d155427f4504ceca4f5a00791005a3b"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "ac05361d97b99d90c105b6f791507762"
server_key_iv: "be193c294ae3d7a091b3099fd0c5989c"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.209450 24575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.222095 24575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:20.224457 24575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:20.224633 24575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:20.224872 24575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:20.225208 24575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:20.225265 24575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.225309 24575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:20.225337 24575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.231444 24575 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:34693
I20260927 04:03:20.231491 24703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:34693 every 8 connection(s)
I20260927 04:03:20.231913 24575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:20.242059 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 24575
I20260927 04:03:20.242182 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:20.242372 18147 external_mini_cluster.cc:1428] Setting key 862f1c37bd93b7baeb2f9cddbb7a5d48
I20260927 04:03:20.242812 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.232768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:44729 (local address 127.17.184.254:34757)
0927 04:03:20.232813 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.232815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.232980 (+ 165us) server_negotiation.cc:407] Connection header received
0927 04:03:20.233512 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.233513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.233524 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.233588 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.235156 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.235703 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.236719 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.236906 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.236967 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.239309 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.239311 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.239311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.239318 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.240985 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.241387 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.241388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.241389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.241443 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.241874 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.241875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.241875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.242041 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.242108 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.242504 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.242627 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:03:20.242964 24706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.232673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:44729)
0927 04:03:20.232844 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.232888 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.233405 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.233657 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.233661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.234736 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.235035 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.235042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.235799 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.235800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.236618 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.236620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.236991 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.236992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.237095 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.237736 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.237745 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.239193 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.241069 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.241074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.241076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.241310 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.241617 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.241618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.241618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.241753 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.242120 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.242127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.242354 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.242482 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.242668 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260927 04:03:20.243399 24704 heartbeater.cc:347] Connected to a master server at 127.17.184.254:34757
I20260927 04:03:20.243516 24704 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:20.243705 24704 heartbeater.cc:511] Master 127.17.184.254:34757 requested a full tablet report, sending...
I20260927 04:03:20.244129 24245 ts_manager.cc:194] Registered new tserver with Master: 5d155427f4504ceca4f5a00791005a3b (127.17.184.195:34693)
I20260927 04:03:20.244519 24245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:44729
I20260927 04:03:20.254733 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.184.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.253.keytab.
Entry for principal oryx/127.17.184.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.253.keytab.
I20260927 04:03:20.282543 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.253:38373
--webserver_interface=127.17.184.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.253.keytab
--principal=oryx/127.17.184.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:34757,127.17.184.253:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:20.364979 24709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:20.365206 24709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:20.365229 24709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:20.366446 24709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:20.366493 24709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:20.366508 24709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:20.366521 24709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:20.366534 24709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:20.368166 24709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:34757,127.17.184.253:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.253.keytab
--principal=oryx/127.17.184.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.253:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.184.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
W20260927 04:03:20.368309 24709 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.17.184.254:34757,127.17.184.253:38373'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 04:03:20.368479 24709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:20.368726 24709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:20.373677 24715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.373699 24717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.373706 24714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:20.373934 24709 server_base.cc:1034] running on GCE node
I20260927 04:03:20.374171 24709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:20.374500 24709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:20.375646 24709 hybrid_clock.cc:648] HybridClock initialized: now 1790481800375637 us; error 31 us; skew 500 ppm
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:20.378336 24709 init.cc:377] Logged in from keytab as oryx/127.17.184.253@KRBTEST.COM (short username oryx)
I20260927 04:03:20.378818 24709 webserver.cc:460] Webserver started at http://127.17.184.253:42661/ using document root <none> and password file <none>
I20260927 04:03:20.379113 24709 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.379182 24709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.379328 24709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:20.380439 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data/instance:
uuid: "0d6696016cb848b08265756e11260656"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "be27ed8514ec0b13bde841438591194b"
server_key_iv: "1b66d6aed91be83c946671d24e7731d6"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.380820 24709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal/instance:
uuid: "0d6696016cb848b08265756e11260656"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "be27ed8514ec0b13bde841438591194b"
server_key_iv: "1b66d6aed91be83c946671d24e7731d6"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.382167 24709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:20.383050 24724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.383340 24709 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:20.383453 24709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
uuid: "0d6696016cb848b08265756e11260656"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "be27ed8514ec0b13bde841438591194b"
server_key_iv: "1b66d6aed91be83c946671d24e7731d6"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.383553 24709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.393158 24709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:20.395841 24709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:20.396020 24709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:20.400094 24709 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.253:38373
I20260927 04:03:20.400135 24776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.253:38373 every 8 connection(s)
I20260927 04:03:20.400490 24709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
I20260927 04:03:20.401571 24777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:20.403205 24777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:03:20.407900 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 24709
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.253@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.184.253@KRBTEST.COM
I20260927 04:03:20.414826 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.404151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.17.184.254:34757)
0927 04:03:20.404244 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.404246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.404483 (+ 237us) server_negotiation.cc:407] Connection header received
0927 04:03:20.405140 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.405141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.405172 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.405232 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.405940 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.406682 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.407951 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.408248 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.408351 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.410982 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.410984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.410984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.410992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.412939 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.413628 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.413629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.413630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.413703 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.414019 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.414020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.414021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.414252 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.414341 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.414640 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.414722 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 04:03:20.415071 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.404079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:39692)
0927 04:03:20.404312 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.404361 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.404980 (+ 619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.405329 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.405333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.405632 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.405847 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.405851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.406791 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.406793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.407818 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.407821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.408362 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.408364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.408542 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.409226 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.409235 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.410856 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.413050 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.413056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.413058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.413443 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.413826 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.413827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.413827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.413942 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.414343 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.414350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.414499 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.414616 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.414792 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":141,"threads_started":1}
I20260927 04:03:20.415678 24777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:03:20.418149 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.408436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54072 (local address 127.17.184.253:38373)
0927 04:03:20.408608 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.408614 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.408639 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:03:20.408688 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.408691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.408712 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.408807 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.409390 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.410230 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.411296 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.411740 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.411803 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.414804 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.414811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.414812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.414820 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.416771 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.417206 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.417210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.417210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.417278 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.417503 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.417504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.417504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.417668 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.417843 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.417938 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.418025 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.253@KRBTEST.COM for oryx/127.17.184.253@KRBTEST.COM
I20260927 04:03:20.423337 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.415995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.0.0.1:54088)
0927 04:03:20.416191 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.416202 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.416263 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.416395 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.416396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.416573 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.416616 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.416617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.417253 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.417254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.417745 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.417746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.418137 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.418138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.418200 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.418670 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.418675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.419955 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.422081 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.422083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.422083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.422249 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.422515 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.422516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.422516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.422577 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.422965 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.422968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.423013 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.423080 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.423167 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260927 04:03:20.423455 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.416056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.17.184.253:38373)
0927 04:03:20.416208 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.416210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.416246 (+ 36us) server_negotiation.cc:407] Connection header received
0927 04:03:20.416317 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.416318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.416328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.416394 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.416693 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.417170 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.417837 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.418056 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.418139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.420039 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.420040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.420040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.420046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.421960 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.422314 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.422314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.422315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.422367 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.422647 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.422648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.422648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.422803 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.422868 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.423108 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.423242 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260927 04:03:20.424826 24777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Bootstrap starting.
I20260927 04:03:20.425408 24777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:20.425653 24777 log.cc:824] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:20.426353 24777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: No bootstrap required, opened a new log
I20260927 04:03:20.427770 24777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } }
I20260927 04:03:20.427887 24777 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:20.427909 24777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d6696016cb848b08265756e11260656, State: Initialized, Role: FOLLOWER
I20260927 04:03:20.428025 24777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } }
I20260927 04:03:20.428277 24777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:20.428279 24784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } } }
I20260927 04:03:20.428418 24784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:03:20.429975 24795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:03:20.430052 24795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:03:20.431540 24709 master_runner.cc:428] Detected that this master 0d6696016cb848b08265756e11260656 is joining an existing cluster
I20260927 04:03:20.431583 24709 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.184.253:38373
I20260927 04:03:20.439378 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.432621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.17.184.253:38373)
0927 04:03:20.432659 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.432661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.432856 (+ 195us) server_negotiation.cc:407] Connection header received
0927 04:03:20.432945 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.432946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.432955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.432997 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.433333 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.433984 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.434823 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.435107 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.435180 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.436457 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.436458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.436459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.436466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.438112 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.438496 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.438497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.438497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.438548 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.438735 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.438736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.438737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.438838 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.438933 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.439100 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.439292 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":121,"server-negotiator.queue_time_us":21}
I20260927 04:03:20.439551 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.432696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.0.0.1:54104)
0927 04:03:20.432802 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.432812 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.432876 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.433009 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.433010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.433198 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.433258 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.433260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.434083 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.434084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.434759 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.434761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.435205 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.435206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.435262 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.435940 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.435944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.436377 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.438231 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.438231 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.438232 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.438417 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.438611 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.438612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.438612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.438665 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.438947 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.438948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.438985 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.439080 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.439237 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":2,"thread_start_us":68,"threads_started":1}
I20260927 04:03:20.440089 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.432550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.17.184.254:34757)
0927 04:03:20.432592 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.432595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.432824 (+ 229us) server_negotiation.cc:407] Connection header received
0927 04:03:20.432906 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.432907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.432918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.432990 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.433662 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.434481 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.435522 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.435807 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.435863 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.436936 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.436937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.436937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.436944 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.438590 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.439001 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.439002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.439003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.439068 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.439284 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.439285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.439285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.439410 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.439452 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.439915 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.439991 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:03:20.440423 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.432602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:39706)
0927 04:03:20.432754 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.432765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.432838 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.433350 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.433351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.433525 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.433578 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.433579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.434580 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.434581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.435435 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.435438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.435899 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.435900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.435991 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.436428 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.436432 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.436831 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.438683 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.438684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.438684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.438888 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.439141 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.439142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.439142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.439197 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.439757 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.439758 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.439800 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.440209 (+ 409us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.440320 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.254@KRBTEST.COM for oryx/127.17.184.253@KRBTEST.COM
I20260927 04:03:20.450736 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.442776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.0.0.1:54108)
0927 04:03:20.442968 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.443012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.443081 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.443236 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.443240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.443483 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.443531 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.443532 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.444449 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.444450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.445067 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.445070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.445572 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.445573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.445642 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.446159 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.446163 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.447735 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.449573 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.449581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.449583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.449810 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.450021 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.450021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.450022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.450078 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.450386 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.450394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.450457 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.450541 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.450647 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":95,"threads_started":1}
I20260927 04:03:20.450762 24782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.442851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.17.184.253:38373)
0927 04:03:20.442919 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.442921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.443082 (+ 161us) server_negotiation.cc:407] Connection header received
0927 04:03:20.443154 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.443155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.443166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.443218 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.443600 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.444336 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.445155 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.445467 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.445563 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.447825 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.447826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.447827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.447834 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.449471 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.449890 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.449891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.449892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.449951 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.450139 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.450140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.450140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.450280 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.450349 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.450588 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.450668 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:03:20.451680 24245 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.184.253:38373: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0d6696016cb848b08265756e11260656"
member_type: NON_VOTER
last_known_addr {
host: "127.17.184.253"
port: 38373
}
attrs {
promote: true
}
}
dest_uuid: "4ba21c2b6e0047109248e6ca90d28c85"
cas_config_opid_index: -1
I20260927 04:03:20.452019 24245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } }
I20260927 04:03:20.452539 24805 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 0d6696016cb848b08265756e11260656 (127.17.184.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } }
I20260927 04:03:20.452847 24751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:20.453598 24805 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.184.253:38373
I20260927 04:03:20.453677 24805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } }
I20260927 04:03:20.453738 24805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:20.453912 24709 master.cc:537] Master@127.17.184.253:38373 shutting down...
I20260927 04:03:20.453953 24751 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Refusing update from remote peer 4ba21c2b6e0047109248e6ca90d28c85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 04:03:20.454303 24804 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 04:03:20.455140 24232 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 04:03:20.456849 24709 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:20.456938 24709 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:20.456988 24709 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: stopping tablet replica
I20260927 04:03:20.459625 24709 master.cc:552] Master@127.17.184.253:38373 shutdown complete.
I20260927 04:03:20.459693 24709 master_runner.cc:305] Clearing existing system tablet
I20260927 04:03:20.459849 24709 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.459915 24709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.460582 24709 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:20.461078 24809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.461206 24709 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.461267 24709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:20.461292 24709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
uuid: "0d6696016cb848b08265756e11260656"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "be27ed8514ec0b13bde841438591194b"
server_key_iv: "1b66d6aed91be83c946671d24e7731d6"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.461894 24709 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:20.463296 24709 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 04:03:20.463361 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.463172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.0.0.1:54110)
0927 04:03:20.463263 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.463319 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 04:03:20.463373 24709 log.cc:1196] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
W20260927 04:03:20.463485 24232 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 -> Peer 0d6696016cb848b08265756e11260656 (127.17.184.253:38373): Couldn't send request to peer 0d6696016cb848b08265756e11260656. Status: Network error: Client connection negotiation failed: client connection to 127.17.184.253:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 04:03:20.463608 24709 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Deleting consensus metadata
I20260927 04:03:20.463719 24709 master_runner.cc:315] Copying system tablet from 127.17.184.254:34757
I20260927 04:03:20.464424 24709 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: tablet copy: Beginning tablet copy session from remote peer at address 127.17.184.254:34757
I20260927 04:03:20.471282 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.464796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.17.184.254:34757)
0927 04:03:20.464841 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.464843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.464913 (+ 70us) server_negotiation.cc:407] Connection header received
0927 04:03:20.464984 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.464985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.464993 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.465044 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.465389 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.465942 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.467039 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.467235 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.467303 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.468445 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.468446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.468446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.468453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.470097 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.470492 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.470493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.470493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.470544 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.470764 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.470764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.470765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.470869 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.470993 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.471150 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.471216 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 04:03:20.471470 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.464701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:39720)
0927 04:03:20.464846 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.464857 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.464922 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.465051 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.465052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.465256 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.465314 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.465316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.466022 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.466023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.466945 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.466948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.467315 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.467316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.467387 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.467898 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.467902 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.468307 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.470180 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.470181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.470182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.470417 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.470607 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.470608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.470608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.470671 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.470985 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.470986 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.471045 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.471126 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.471240 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260927 04:03:20.471853 24265 tablet_copy_service.cc:140] P 4ba21c2b6e0047109248e6ca90d28c85: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0d6696016cb848b08265756e11260656 ({username='oryx', principal='oryx/127.17.184.253@KRBTEST.COM'} at 127.0.0.1:39720)
I20260927 04:03:20.471954 24265 tablet_copy_service.cc:161] P 4ba21c2b6e0047109248e6ca90d28c85: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0d6696016cb848b08265756e11260656 at {username='oryx', principal='oryx/127.17.184.253@KRBTEST.COM'} at 127.0.0.1:39720: session id = 0d6696016cb848b08265756e11260656-00000000000000000000000000000000
I20260927 04:03:20.472903 24265 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 04:03:20.473614 24709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 04:03:20.474823 24709 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: tablet copy: Starting download of 0 data blocks...
I20260927 04:03:20.475075 24709 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: tablet copy: Starting download of 1 WAL segments...
I20260927 04:03:20.476137 24709 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 04:03:20.476994 24265 tablet_copy_service.cc:342] P 4ba21c2b6e0047109248e6ca90d28c85: Request end of tablet copy session 0d6696016cb848b08265756e11260656-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.184.253@KRBTEST.COM'} at 127.0.0.1:39720
I20260927 04:03:20.477061 24265 tablet_copy_service.cc:433] P 4ba21c2b6e0047109248e6ca90d28c85: ending tablet copy session 0d6696016cb848b08265756e11260656-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0d6696016cb848b08265756e11260656
I20260927 04:03:20.479589 24709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:20.480937 24817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.481051 24819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.481113 24816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:20.481310 24709 server_base.cc:1034] running on GCE node
I20260927 04:03:20.481416 24709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:20.481591 24709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:20.482782 24709 hybrid_clock.cc:648] HybridClock initialized: now 1790481800482774 us; error 25 us; skew 500 ppm
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:20.484320 24709 init.cc:377] Logged in from keytab as oryx/127.17.184.253@KRBTEST.COM (short username oryx)
I20260927 04:03:20.484587 24709 webserver.cc:460] Webserver started at http://127.17.184.253:39565/ using document root <none> and password file <none>
I20260927 04:03:20.484709 24709 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.484745 24709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.485488 24709 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:20.486102 24826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.486294 24709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:20.486347 24709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
uuid: "0d6696016cb848b08265756e11260656"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "be27ed8514ec0b13bde841438591194b"
server_key_iv: "1b66d6aed91be83c946671d24e7731d6"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.486406 24709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.523839 24709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:20.525645 24709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:20.528839 24709 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.253:38373
I20260927 04:03:20.529022 24878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.253:38373 every 8 connection(s)
I20260927 04:03:20.529209 24709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
I20260927 04:03:20.530148 24879 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:03:20.530634 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Bootstrap starting.
I20260927 04:03:20.533483 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:03:20.533727 24879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: Bootstrap complete.
I20260927 04:03:20.533941 24879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } }
I20260927 04:03:20.534029 24879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0d6696016cb848b08265756e11260656, State: Initialized, Role: LEARNER
I20260927 04:03:20.534070 24879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } }
I20260927 04:03:20.534276 24879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:20.534264 24882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } }
I20260927 04:03:20.534401 24882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:03:20.535816 24893 catalog_manager.cc:1259] Loaded cluster ID: c86a0064840946bcb8825eaa90b1706f
I20260927 04:03:20.535882 24893 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: loading cluster ID for follower catalog manager: success
I20260927 04:03:20.537081 24893 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: acquiring CA information for follower catalog manager: success
I20260927 04:03:20.537374 24893 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:03:20.577680 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.570700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54152 (local address 127.17.184.253:38373)
0927 04:03:20.570841 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.570843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.570850 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:03:20.570872 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.570873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.570883 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.570990 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.571584 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.572371 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.573201 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.573435 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.573519 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.574807 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.574808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.574809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.574816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.576437 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.576803 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.576804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.576805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.576867 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.577066 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.577067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.577067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.577188 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.577292 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.577441 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.577530 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
WARNING: no policy specified for oryx/127.17.184.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.252.keytab.
Entry for principal oryx/127.17.184.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.252.keytab.
I20260927 04:03:20.606879 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.252:36099
--webserver_interface=127.17.184.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.252.keytab
--principal=oryx/127.17.184.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:34757,127.17.184.253:38373,127.17.184.252:36099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:20.694944 24897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:20.695209 24897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:20.695240 24897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:20.696439 24897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:20.696482 24897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:20.696497 24897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:20.696509 24897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:20.696522 24897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:20.698060 24897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:34757,127.17.184.253:38373,127.17.184.252:36099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.252.keytab
--principal=oryx/127.17.184.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.252:36099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.184.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:20.698310 24897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:20.698555 24897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:20.703634 24903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:20.703670 24897 server_base.cc:1034] running on GCE node
W20260927 04:03:20.703649 24905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.703812 24902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:20.704190 24897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:20.704523 24897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:20.705660 24897 hybrid_clock.cc:648] HybridClock initialized: now 1790481800705638 us; error 40 us; skew 500 ppm
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:20.708245 24897 init.cc:377] Logged in from keytab as oryx/127.17.184.252@KRBTEST.COM (short username oryx)
I20260927 04:03:20.708760 24897 webserver.cc:460] Webserver started at http://127.17.184.252:35793/ using document root <none> and password file <none>
I20260927 04:03:20.709005 24897 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.709072 24897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.709218 24897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:20.710285 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data/instance:
uuid: "ae095317e0aa47fb85ea3292e86dfe17"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "c891d83e16355d0269b0fb6f4570fe85"
server_key_iv: "67fc06e7ade8871e9ff0253b949219ed"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.710685 24897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal/instance:
uuid: "ae095317e0aa47fb85ea3292e86dfe17"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "c891d83e16355d0269b0fb6f4570fe85"
server_key_iv: "67fc06e7ade8871e9ff0253b949219ed"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.712097 24897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:20.712836 24912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.713034 24897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:20.713135 24897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
uuid: "ae095317e0aa47fb85ea3292e86dfe17"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "c891d83e16355d0269b0fb6f4570fe85"
server_key_iv: "67fc06e7ade8871e9ff0253b949219ed"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.713223 24897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.732335 24897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:20.734941 24897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:20.735116 24897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:20.739693 24897 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.252:36099
I20260927 04:03:20.739753 24964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.252:36099 every 8 connection(s)
I20260927 04:03:20.740146 24897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
I20260927 04:03:20.741178 24965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:20.742108 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 24897
I20260927 04:03:20.743459 24965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.252@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.184.252@KRBTEST.COM
I20260927 04:03:20.752264 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.744381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.17.184.254:34757)
0927 04:03:20.744453 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.744456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.744484 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:03:20.744567 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.744568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.744579 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.744650 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.745151 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.745634 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.746518 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.746712 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.746806 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.749006 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.749007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.749007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.749014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.750857 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.751284 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.751285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.751285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.751348 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.751594 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.751596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.751596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.751728 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.751812 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.752086 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.752166 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 04:03:20.752487 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.744192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:39732)
0927 04:03:20.744370 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.744430 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.744503 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.744671 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.744674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.744972 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.745058 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.745062 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.745745 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.745746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.746427 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.746428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.746836 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.746838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.746955 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.747459 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.747466 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.748905 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.750971 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.750976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.750978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.751205 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.751419 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.751420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.751420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.751516 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.751812 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.751819 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.751947 (+ 128us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.752052 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.752223 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260927 04:03:20.753021 24965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:03:20.753155 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.742921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60048 (local address 127.17.184.252:36099)
0927 04:03:20.743090 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.743099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.743168 (+ 69us) server_negotiation.cc:407] Connection header received
0927 04:03:20.743612 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.743619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.743676 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.743830 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.744601 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.745475 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.746213 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.746565 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.746702 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.749593 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.749600 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.749601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.749607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.751450 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.751791 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.751794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.751794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.751850 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.752057 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.752058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.752058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.752201 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.752868 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.752915 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.753015 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.252@KRBTEST.COM for oryx/127.17.184.253@KRBTEST.COM
I20260927 04:03:20.760252 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.753462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.0.0.1:54160)
0927 04:03:20.753577 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.753592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.753668 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.753862 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.753863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.753973 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.754015 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.754016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.754849 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.754850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.755473 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.755476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.755855 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.755856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.755905 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.756366 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.756370 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.757560 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.759284 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.759284 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.759285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.759449 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.759640 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.759640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.759641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.759773 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.759997 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.759998 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.760035 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.760088 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.760175 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 04:03:20.760257 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.753532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.17.184.253:38373)
0927 04:03:20.753583 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.753585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.753657 (+ 72us) server_negotiation.cc:407] Connection header received
0927 04:03:20.753729 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.753730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.753785 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.753853 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.754141 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.754761 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.755553 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.755788 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.755845 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.757635 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.757643 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.757644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.757650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.759189 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.759516 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.759516 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.759517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.759564 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.759839 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.759839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.759840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.759940 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.759991 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.760125 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.760191 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:03:20.760838 24965 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.252@KRBTEST.COM for oryx/127.17.184.252@KRBTEST.COM
I20260927 04:03:20.767334 24968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.761143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.0.0.1:60064)
0927 04:03:20.761244 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.761257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.761315 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.761432 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.761432 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.761550 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.761583 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.761584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.762240 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.762241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.762770 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.762771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.763261 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.763262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.763306 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.763754 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.763758 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.764923 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.766393 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.766394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.766394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.766553 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.766759 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.766760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.766760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.766810 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.767078 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.767078 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.767122 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.767178 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.767267 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 04:03:20.767477 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.761251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.17.184.252:36099)
0927 04:03:20.761309 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.761310 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.761317 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:03:20.761377 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.761378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.761385 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.761427 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.761641 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.762107 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.762861 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.763142 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.763204 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.765005 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.765007 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.765007 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.765013 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.766311 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.766618 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.766619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.766619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.766666 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.766866 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.766867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.766867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.767019 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.767072 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.767207 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.767332 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":42}
I20260927 04:03:20.768805 24965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Bootstrap starting.
I20260927 04:03:20.769644 24965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:20.769936 24965 log.cc:824] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:20.770694 24965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: No bootstrap required, opened a new log
I20260927 04:03:20.772203 24965 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } }
I20260927 04:03:20.772341 24965 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:20.772367 24965 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae095317e0aa47fb85ea3292e86dfe17, State: Initialized, Role: FOLLOWER
I20260927 04:03:20.772498 24965 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } }
I20260927 04:03:20.772819 24965 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:20.772784 24970 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } } }
I20260927 04:03:20.772893 24970 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:03:20.774592 24981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:03:20.774645 24981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:03:20.777064 24897 master_runner.cc:428] Detected that this master ae095317e0aa47fb85ea3292e86dfe17 is joining an existing cluster
I20260927 04:03:20.777114 24897 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.184.252:36099
I20260927 04:03:20.785970 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.778225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:39738)
0927 04:03:20.778408 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.778420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.778512 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.778720 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.778721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.779006 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.779093 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.779095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.780202 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.780202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.781250 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.781253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.781795 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.781877 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.782460 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.782465 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.782849 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.784854 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.784854 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.784855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.785047 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.785265 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.785265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.785266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.785323 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.785659 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.785660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.785702 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.785781 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.785877 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":26,"thread_start_us":90,"threads_started":1}
I20260927 04:03:20.786006 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.778299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39738 (local address 127.17.184.254:34757)
0927 04:03:20.778381 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.778383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.778476 (+ 93us) server_negotiation.cc:407] Connection header received
0927 04:03:20.778616 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.778617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.778629 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.778698 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.779206 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.780090 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781354 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.781679 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781719 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.783050 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.783051 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.783051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.783059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.784754 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.785127 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.785128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.785129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.785193 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.785389 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.785390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.785390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.785563 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.785636 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.785824 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.785903 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481799, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.254@KRBTEST.COM for oryx/127.17.184.252@KRBTEST.COM
I20260927 04:03:20.802428 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.788199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.0.0.1:60084)
0927 04:03:20.788268 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.788281 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.788324 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.788710 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.788711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.788850 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.788888 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.788889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.789591 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.789592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.790124 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.790126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.790510 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.790510 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.790583 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.791028 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.791040 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.792425 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.801166 (+ 8741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.801168 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.801169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.801410 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.801644 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.801645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.801646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.801719 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.802040 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.802041 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.802101 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.802154 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.802281 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 04:03:20.802831 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.788385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.17.184.252:36099)
0927 04:03:20.788569 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.788570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.788576 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:03:20.788604 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.788605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.788619 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.788693 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.788968 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.789501 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.790234 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.790439 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.790513 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.792561 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.792563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.792563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.792571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.801037 (+ 8466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.801504 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.801505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.801505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.801565 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.801800 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.801800 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.801801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.801949 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.802559 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.802611 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.802702 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260927 04:03:20.803025 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.778526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54168 (local address 127.17.184.253:38373)
0927 04:03:20.778563 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.778565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.778840 (+ 275us) server_negotiation.cc:407] Connection header received
0927 04:03:20.778975 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.778976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.778986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.779016 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.779482 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.780253 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781353 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.781644 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781686 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.800065 (+ 18379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.800067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.800068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.800077 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.801660 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.802060 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.802061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.802061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.802115 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.802338 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.802339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.802340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.802473 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.802516 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.802805 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.802890 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:03:20.803627 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.778505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.0.0.1:54168)
0927 04:03:20.778783 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.778794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.778864 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.779080 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.779080 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.779327 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.779391 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.779392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.780352 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.780352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.781253 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.781255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.781773 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.781862 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.799412 (+ 17550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.799418 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.799924 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.801763 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.801764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.801764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.801975 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.802185 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.802186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.802186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.802263 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.802645 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.802646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.802691 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.803396 (+ 705us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.803512 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260927 04:03:20.804016 24245 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.184.252:36099: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17"
member_type: NON_VOTER
last_known_addr {
host: "127.17.184.252"
port: 36099
}
attrs {
promote: true
}
}
dest_uuid: "4ba21c2b6e0047109248e6ca90d28c85"
cas_config_opid_index: 5
I20260927 04:03:20.804466 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 24307
I20260927 04:03:20.804421 24245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } }
I20260927 04:03:20.805191 24804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.806088 24804 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.184.252:36099
I20260927 04:03:20.806185 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.806269 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:20.806353 24897 master.cc:537] Master@127.17.184.252:36099 shutting down...
W20260927 04:03:20.808220 24229 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 -> Peer ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252:36099): Couldn't send request to peer ae095317e0aa47fb85ea3292e86dfe17. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 04:03:20.809017 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.778244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.0.0.1:60072)
0927 04:03:20.778716 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.778727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.778802 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.779029 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.779030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.779277 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.779343 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.779345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.780274 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.780274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.781252 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.781254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.781796 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.781877 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.801152 (+ 19275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.801157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.801550 (+ 393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.803968 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.803969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.803969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.804193 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.808218 (+ 4025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.808218 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.808219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.808273 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.808582 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.808583 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.808643 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.808720 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.808841 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":46,"thread_start_us":74,"threads_started":1}
I20260927 04:03:20.809576 24967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.778340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.17.184.252:36099)
0927 04:03:20.778409 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.778411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.778779 (+ 368us) server_negotiation.cc:407] Connection header received
0927 04:03:20.778868 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.778869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.778877 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.778953 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.779422 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.780191 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781352 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.781665 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.781706 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.801638 (+ 19932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.801639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.801639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.801646 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.803887 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.808079 (+ 4192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.808080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.808080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.808147 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.808363 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.808364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.808364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.808496 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.808552 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.809330 (+ 778us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.809412 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 04:03:20.811231 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:33249
--local_ip_for_outbound_sockets=127.17.184.193
--tserver_master_addrs=127.17.184.254:34757
--webserver_port=41321
--webserver_interface=127.17.184.193
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab
--principal=oryx/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373,127.17.184.252:36099 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:03:20.812567 24897 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:20.812669 24897 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:20.812714 24897 tablet_replica.cc:330] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: stopping tablet replica
I20260927 04:03:20.813680 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.807476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.0.0.1:54174)
0927 04:03:20.807898 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.807909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.807946 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.810046 (+ 2100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.810047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.810049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:20.810104 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.810105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.811070 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.811071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.812357 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.812360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.813394 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.813396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.813463 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.813488 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.813543 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.813597 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405}
I20260927 04:03:20.814010 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.808656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.17.184.253:38373)
0927 04:03:20.808699 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.808700 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.808707 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:03:20.808739 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.808740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.808764 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.808824 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:20.810201 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.810984 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.812480 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.813174 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.813249 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.813334 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.813871 (+ 537us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.813921 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:03:20.814227 24853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 LEARNER]: Refusing update from remote peer 4ba21c2b6e0047109248e6ca90d28c85: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 04:03:20.814411 24805 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 04:03:20.815091 24853 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.816744 24882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ba21c2b6e0047109248e6ca90d28c85. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.816875 24882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:03:20.816735 24993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: NON_VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: true } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.817000 24993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:03:20.817407 24805 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: attempting to promote NON_VOTER 0d6696016cb848b08265756e11260656 to VOTER
I20260927 04:03:20.817669 24805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } }
I20260927 04:03:20.818399 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.818279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.0.0.1:60096)
0927 04:03:20.818327 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.818360 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 04:03:20.818444 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.818309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.0.0.1:60106)
0927 04:03:20.818375 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.818409 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 04:03:20.818629 24853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 LEARNER]: Refusing update from remote peer 4ba21c2b6e0047109248e6ca90d28c85: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 04:03:20.819113 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.819015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.0.0.1:60116)
0927 04:03:20.819055 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.819079 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:36099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 04:03:20.819188 24229 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 -> Peer ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252:36099): Couldn't send request to peer ae095317e0aa47fb85ea3292e86dfe17. Status: Network error: Client connection negotiation failed: client connection to 127.17.184.252:36099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 04:03:20.819250 24994 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 04:03:20.820017 24804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0d6696016cb848b08265756e11260656 (127.17.184.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.821182 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.821271 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:20.821549 24805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.821628 24805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:20.821962 24853 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 0d6696016cb848b08265756e11260656 (127.17.184.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.823189 24993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.823289 24993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:03:20.827538 24897 master.cc:552] Master@127.17.184.252:36099 shutdown complete.
I20260927 04:03:20.827592 24897 master_runner.cc:305] Clearing existing system tablet
I20260927 04:03:20.827729 24897 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.827775 24897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.828519 24897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:20.829099 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.829214 24897 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.829262 24897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.829285 24897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
uuid: "ae095317e0aa47fb85ea3292e86dfe17"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "c891d83e16355d0269b0fb6f4570fe85"
server_key_iv: "67fc06e7ade8871e9ff0253b949219ed"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.830118 24897 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:20.831643 24897 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 04:03:20.831719 24897 log.cc:1196] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 04:03:20.831916 24897 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Deleting consensus metadata
I20260927 04:03:20.832185 24897 master_runner.cc:315] Copying system tablet from 127.17.184.254:34757
I20260927 04:03:20.832790 24897 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: tablet copy: Beginning tablet copy session from remote peer at address 127.17.184.254:34757
I20260927 04:03:20.850582 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.843155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:39742)
0927 04:03:20.843333 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.843346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.843425 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.843590 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.843591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.843838 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.843911 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.843913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.844864 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.844865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.845876 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.845878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.846343 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.846344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.846414 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.847172 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.847177 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.847588 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.849397 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.849398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.849398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.849627 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.849869 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.849870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.849870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.849936 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.850235 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.850236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.850285 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.850361 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.850471 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260927 04:03:20.850596 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.843237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.17.184.254:34757)
0927 04:03:20.843294 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.843297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.843424 (+ 127us) server_negotiation.cc:407] Connection header received
0927 04:03:20.843496 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.843498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.843510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.843574 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.843992 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.844764 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.845966 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.846259 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.846331 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.847682 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.847684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.847684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.847691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.849317 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.849716 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.849718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.849718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.849794 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.850005 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.850006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.850006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.850158 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.850237 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.850406 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.850488 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 04:03:20.851248 24265 tablet_copy_service.cc:140] P 4ba21c2b6e0047109248e6ca90d28c85: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ae095317e0aa47fb85ea3292e86dfe17 ({username='oryx', principal='oryx/127.17.184.252@KRBTEST.COM'} at 127.0.0.1:39742)
I20260927 04:03:20.851301 24265 tablet_copy_service.cc:161] P 4ba21c2b6e0047109248e6ca90d28c85: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ae095317e0aa47fb85ea3292e86dfe17 at {username='oryx', principal='oryx/127.17.184.252@KRBTEST.COM'} at 127.0.0.1:39742: session id = ae095317e0aa47fb85ea3292e86dfe17-00000000000000000000000000000000
I20260927 04:03:20.851892 24265 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 04:03:20.852342 24897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 04:03:20.853717 24897 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: tablet copy: Starting download of 0 data blocks...
I20260927 04:03:20.853869 24897 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: tablet copy: Starting download of 1 WAL segments...
I20260927 04:03:20.854664 24897 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 04:03:20.855562 24265 tablet_copy_service.cc:342] P 4ba21c2b6e0047109248e6ca90d28c85: Request end of tablet copy session ae095317e0aa47fb85ea3292e86dfe17-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.184.252@KRBTEST.COM'} at 127.0.0.1:39742
I20260927 04:03:20.855625 24265 tablet_copy_service.cc:433] P 4ba21c2b6e0047109248e6ca90d28c85: ending tablet copy session ae095317e0aa47fb85ea3292e86dfe17-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ae095317e0aa47fb85ea3292e86dfe17
I20260927 04:03:20.858520 24897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:20.859740 25007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.859750 25004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.860332 25005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:20.863431 24897 server_base.cc:1034] running on GCE node
I20260927 04:03:20.863548 24897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:20.863711 24897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:20.864847 24897 hybrid_clock.cc:648] HybridClock initialized: now 1790481800864836 us; error 30 us; skew 500 ppm
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:20.866452 24897 init.cc:377] Logged in from keytab as oryx/127.17.184.252@KRBTEST.COM (short username oryx)
I20260927 04:03:20.866758 24897 webserver.cc:460] Webserver started at http://127.17.184.252:39137/ using document root <none> and password file <none>
I20260927 04:03:20.866885 24897 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.866951 24897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.867738 24897 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:20.868338 25014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.868511 24897 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:20.868568 24897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
uuid: "ae095317e0aa47fb85ea3292e86dfe17"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "c891d83e16355d0269b0fb6f4570fe85"
server_key_iv: "67fc06e7ade8871e9ff0253b949219ed"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.868633 24897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.881372 24897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:20.883332 24897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:20.886468 24897 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.252:36099
I20260927 04:03:20.886911 24897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
I20260927 04:03:20.887305 25066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.252:36099 every 8 connection(s)
I20260927 04:03:20.888082 25067 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:03:20.888618 25067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Bootstrap starting.
I20260927 04:03:20.891790 25067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:03:20.892047 25067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Bootstrap complete.
I20260927 04:03:20.892275 25067 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } }
I20260927 04:03:20.892386 25067 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ae095317e0aa47fb85ea3292e86dfe17, State: Initialized, Role: LEARNER
I20260927 04:03:20.892426 25067 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } }
I20260927 04:03:20.892625 25067 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:20.894150 25081 catalog_manager.cc:1259] Loaded cluster ID: c86a0064840946bcb8825eaa90b1706f
I20260927 04:03:20.894249 25081 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: loading cluster ID for follower catalog manager: success
I20260927 04:03:20.895249 25070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:20.895356 25070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: This master's current role is: LEARNER
I20260927 04:03:20.895740 25081 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: acquiring CA information for follower catalog manager: success
I20260927 04:03:20.896131 25081 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 04:03:20.915608 24991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:20.915863 24991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:20.915904 24991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:20.917254 24991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:20.917317 24991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:20.917364 24991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:20.918984 24991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab
--principal=oryx/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:33249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=41321
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373,127.17.184.252:36099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:20.919258 24991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:20.919519 24991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:20.925908 25086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.925880 25087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:20.925877 25089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:20.926236 24991 server_base.cc:1034] running on GCE node
I20260927 04:03:20.926468 24991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:20.926723 24991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:20.928174 24991 hybrid_clock.cc:648] HybridClock initialized: now 1790481800928137 us; error 52 us; skew 500 ppm
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:20.930708 24991 init.cc:377] Logged in from keytab as oryx/127.17.184.193@KRBTEST.COM (short username oryx)
I20260927 04:03:20.931183 24991 webserver.cc:460] Webserver started at http://127.17.184.193:41321/ using document root <none> and password file <none>
I20260927 04:03:20.931432 24991 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:20.931522 24991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:20.932961 24991 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:20.933637 25096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.933797 24991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:20.933921 24991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "60d807011b38409097e30e028fb20cbb"
format_stamp: "Formatted at 2026-09-27 04:03:19 on dist-test-slave-zqrn"
server_key: "36ada9b134d63c3d1789153e1eb6f67a"
server_key_iv: "3dc3af73021e6a6b8bc5e90622242cd1"
server_key_version: "encryptionkey@0"
I20260927 04:03:20.934314 24991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:20.945899 24991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:20.948213 24991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:20.948400 24991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:20.948653 24991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:20.948969 24991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:20.949024 24991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.949060 24991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:20.949107 24991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:20.955379 24991 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:33249
I20260927 04:03:20.955459 25209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:33249 every 8 connection(s)
I20260927 04:03:20.955833 24991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:20.957826 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 24991
I20260927 04:03:20.957978 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 24441
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:20.964128 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:45969
--local_ip_for_outbound_sockets=127.17.184.194
--tserver_master_addrs=127.17.184.254:34757
--webserver_port=45577
--webserver_interface=127.17.184.194
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab
--principal=oryx/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373,127.17.184.252:36099 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.252@KRBTEST.COM
Sep 27 04:03:20 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481800, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.253@KRBTEST.COM
I20260927 04:03:20.968953 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.956816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:45053 (local address 127.17.184.254:34757)
0927 04:03:20.956862 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.956864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.956969 (+ 105us) server_negotiation.cc:407] Connection header received
0927 04:03:20.959813 (+ 2844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.959815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.959828 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.959900 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.960258 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.960736 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.961645 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.961845 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.961931 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.965413 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.965415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.965415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.965423 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.967222 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.967794 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.967795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.967796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.967854 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.968149 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.968151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.968151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.968308 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.968353 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.968775 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.968859 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:03:20.969151 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.956567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.17.184.193:46979)
0927 04:03:20.956750 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.956794 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.957932 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.958325 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.958329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.958752 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.959071 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.959077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.960513 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.960514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.961780 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.961791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.962362 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.962363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.962431 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.962854 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.962859 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.965672 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.967689 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.967690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.967690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.967913 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.968168 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.968168 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.968169 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.968225 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.968548 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.968549 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.968593 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.968711 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.968896 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260927 04:03:20.969244 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.956568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:45053)
0927 04:03:20.956916 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.956925 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.959711 (+ 2786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.959921 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.959922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.960091 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.960150 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.960151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.960820 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.960821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.961538 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.961540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.961942 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.961943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.962070 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.962629 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.962640 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.965305 (+ 2665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.967318 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.967327 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.967330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.967683 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.967928 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.967929 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.967929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.968068 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.968400 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.968406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.968578 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.968697 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.968990 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":123,"thread_start_us":78,"threads_started":1}
I20260927 04:03:20.969596 25211 heartbeater.cc:347] Connected to a master server at 127.17.184.254:34757
I20260927 04:03:20.969755 25211 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:20.969960 25211 heartbeater.cc:511] Master 127.17.184.254:34757 requested a full tablet report, sending...
I20260927 04:03:20.970463 24245 ts_manager.cc:194] Re-registered known tserver with Master: 60d807011b38409097e30e028fb20cbb (127.17.184.193:33249)
I20260927 04:03:20.970674 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.956601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:46979 (local address 127.17.184.252:36099)
0927 04:03:20.957087 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.957088 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.957096 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:20.958102 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.958104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.958119 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.958181 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.959172 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.960377 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.961888 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.962248 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.962310 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.965840 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.965841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.965841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.965849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.967598 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.967996 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.967997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.967998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.968069 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.968310 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.968311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.968312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.968454 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.968520 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.970458 (+ 1938us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.970558 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":61,"threads_started":1}
I20260927 04:03:20.971040 25210 heartbeater.cc:347] Connected to a master server at 127.17.184.252:36099
I20260927 04:03:20.971088 25210 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:20.971069 24245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:45053
I20260927 04:03:20.971161 25210 heartbeater.cc:511] Master 127.17.184.252:36099 requested a full tablet report, sending...
I20260927 04:03:20.971539 25023 ts_manager.cc:194] Registered new tserver with Master: 60d807011b38409097e30e028fb20cbb (127.17.184.193:33249)
I20260927 04:03:20.971997 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.956667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:48257 (local address 127.17.184.253:38373)
0927 04:03:20.956714 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:20.956717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:20.957030 (+ 313us) server_negotiation.cc:407] Connection header received
0927 04:03:20.958049 (+ 1019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:20.958050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:20.958063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:20.958117 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:20.963841 (+ 5724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.964459 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.965234 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.965499 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.965562 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.967774 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:20.967776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:20.967776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:20.967784 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:20.970202 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.970660 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.970661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.970662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.970720 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:20.971346 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:20.971348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:20.971348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:20.971510 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:20.971593 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:20.971825 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 04:03:20.971906 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:03:20.972226 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:20.956567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.17.184.193:48257)
0927 04:03:20.956959 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:20.956970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:20.957960 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:20.963524 (+ 5564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:20.963526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:20.963686 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:20.963746 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.963747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.964556 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.964557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.965151 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:20.965152 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:20.965621 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:20.965622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:20.965693 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:20.966155 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:20.966159 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:20.967670 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:20.970283 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.970285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.970285 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.970496 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.970926 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:20.970927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:20.970927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:20.970990 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:20.971587 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:20.971587 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:20.971646 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 04:03:20.971729 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 04:03:20.971838 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":221,"thread_start_us":56,"threads_started":1}
I20260927 04:03:20.972695 25212 heartbeater.cc:347] Connected to a master server at 127.17.184.253:38373
I20260927 04:03:20.972761 25212 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:20.972836 25212 heartbeater.cc:511] Master 127.17.184.253:38373 requested a full tablet report, sending...
I20260927 04:03:20.973203 24843 ts_manager.cc:194] Registered new tserver with Master: 60d807011b38409097e30e028fb20cbb (127.17.184.193:33249)
W20260927 04:03:21.055599 25218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:21.055820 25218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:21.055846 25218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:21.057101 25218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:21.057147 25218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:21.057173 25218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:21.058687 25218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab
--principal=oryx/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:45969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=45577
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373,127.17.184.252:36099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:21.058966 25218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:21.059218 25218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:21.064386 25223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:21.064386 25226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:21.064445 25218 server_base.cc:1034] running on GCE node
W20260927 04:03:21.064365 25224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:21.064870 25218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:21.065081 25218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:21.066220 25218 hybrid_clock.cc:648] HybridClock initialized: now 1790481801066193 us; error 40 us; skew 500 ppm
Sep 27 04:03:21 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481801, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:21.069133 25218 init.cc:377] Logged in from keytab as oryx/127.17.184.194@KRBTEST.COM (short username oryx)
I20260927 04:03:21.069557 25218 webserver.cc:460] Webserver started at http://127.17.184.194:45577/ using document root <none> and password file <none>
I20260927 04:03:21.069859 25218 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:21.069917 25218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:21.071347 25218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 04:03:21.072011 25233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:21.072206 25218 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:21.072299 25218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "e84555cae7414ffc947a877ac4315649"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "0a79483e90e67f5a2e8c30f486e80d6d"
server_key_iv: "cdfd8b7acd86bc6d69b1afef39233928"
server_key_version: "encryptionkey@0"
I20260927 04:03:21.072643 25218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:21.094277 25218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:21.096714 25218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:21.096886 25218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:21.097126 25218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:21.097442 25218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:21.097496 25218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:21.097537 25218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:21.097565 25218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:21.104169 25218 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:45969
I20260927 04:03:21.104241 25346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:45969 every 8 connection(s)
I20260927 04:03:21.104717 25218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:21.112407 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 25218
I20260927 04:03:21.112586 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 24575
Sep 27 04:03:21 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481801, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.253@KRBTEST.COM
Sep 27 04:03:21 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481801, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.252@KRBTEST.COM
Sep 27 04:03:21 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481801, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:21.118768 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.105630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:39163 (local address 127.17.184.254:34757)
0927 04:03:21.105711 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.105714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.106096 (+ 382us) server_negotiation.cc:407] Connection header received
0927 04:03:21.106723 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.106725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.106740 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.106776 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.108272 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.108862 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.110388 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.110724 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.110769 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.114873 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.114875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.114876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.114884 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.117504 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117942 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.117943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.117943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.117999 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.118228 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.118229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.118230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.118354 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.118437 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.118600 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.118675 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:03:21.118768 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.105870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:33929 (local address 127.17.184.253:38373)
0927 04:03:21.105989 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.105992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.106004 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:03:21.106727 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.106729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.106741 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.106776 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.107984 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.108751 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.110383 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.110599 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.110691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.115045 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.115046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.115046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.115054 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.116996 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117479 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.117480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.117480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.117539 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117845 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.117847 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.117847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.118009 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.118053 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.118551 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.118650 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260927 04:03:21.119091 25351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.105591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.17.184.194:33929)
0927 04:03:21.105785 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.105832 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.106581 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.107002 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.107007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.107561 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:21.107889 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.107894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.109708 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.109709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.110287 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.110289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.110696 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.110697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.110848 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.111614 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:21.111622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:21.114932 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:21.117090 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.117091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.117396 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.117620 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.117622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.117768 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.118096 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:21.118103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:21.118393 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.118528 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.118721 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":45,"spinlock_wait_cycles":8576,"thread_start_us":105,"threads_started":1}
I20260927 04:03:21.119196 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.105591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:39163)
0927 04:03:21.106037 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.106047 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.106591 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.107422 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.107423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.108117 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:21.108177 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.108179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.108980 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.108981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.110280 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.110282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.110848 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.110849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.110941 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.111907 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:21.111912 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:21.114737 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:21.117595 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.117597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.117860 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.118087 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.118088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.118088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.118144 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.118440 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:21.118442 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:21.118484 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.118920 (+ 436us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.119058 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":305,"thread_start_us":61,"threads_started":1}
I20260927 04:03:21.119558 25348 heartbeater.cc:347] Connected to a master server at 127.17.184.254:34757
I20260927 04:03:21.119563 25349 heartbeater.cc:347] Connected to a master server at 127.17.184.253:38373
I20260927 04:03:21.119673 25349 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:21.119891 25349 heartbeater.cc:511] Master 127.17.184.253:38373 requested a full tablet report, sending...
I20260927 04:03:21.119673 25348 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:21.120157 25348 heartbeater.cc:511] Master 127.17.184.254:34757 requested a full tablet report, sending...
I20260927 04:03:21.120275 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:34693
--local_ip_for_outbound_sockets=127.17.184.195
--tserver_master_addrs=127.17.184.254:34757
--webserver_port=40753
--webserver_interface=127.17.184.195
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab
--principal=oryx/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373,127.17.184.252:36099 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:03:21.120375 24843 ts_manager.cc:194] Registered new tserver with Master: e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969)
I20260927 04:03:21.120592 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.105934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:51111 (local address 127.17.184.252:36099)
0927 04:03:21.106038 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.106040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.106048 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:21.106766 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.106767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.106786 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.106839 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.108866 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.109578 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.110721 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.111110 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.111159 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.114914 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.114915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.114915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.114923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.116863 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117463 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.117465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.117465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.117529 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117864 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.117866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.117866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.118022 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.118068 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.120385 (+ 2317us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.120483 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 04:03:21.120756 25352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.105591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.17.184.194:51111)
0927 04:03:21.105807 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.105832 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.106586 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.107024 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.107025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.107561 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:21.107889 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.107894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.109686 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.109688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.110612 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.110614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.111243 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.111244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.111312 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.111768 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:21.111774 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:21.114740 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:21.116980 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.116988 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.116991 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.117362 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.117611 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.117612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.117612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.117765 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.118114 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:21.118116 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:21.120246 (+ 2130us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.120354 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.120461 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1426,"thread_start_us":120,"threads_started":1}
I20260927 04:03:21.121841 24245 ts_manager.cc:194] Re-registered known tserver with Master: e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969)
I20260927 04:03:21.122229 24245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:39163
I20260927 04:03:21.122368 25347 heartbeater.cc:347] Connected to a master server at 127.17.184.252:36099
I20260927 04:03:21.122416 25347 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:21.122479 25347 heartbeater.cc:511] Master 127.17.184.252:36099 requested a full tablet report, sending...
I20260927 04:03:21.122793 25023 ts_manager.cc:194] Registered new tserver with Master: e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969)
W20260927 04:03:21.204205 25354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:21.204448 25354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:21.204488 25354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:21.205780 25354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:21.205843 25354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:21.205890 25354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:21.207525 25354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab
--principal=oryx/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:34693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=40753
--tserver_master_addrs=127.17.184.254:34757,127.17.184.253:38373,127.17.184.252:36099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:21.207808 25354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:21.208043 25354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:21.213160 25359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:21.213181 25360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:21.213192 25362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:21.213254 25354 server_base.cc:1034] running on GCE node
I20260927 04:03:21.213579 25354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:21.213836 25354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:21.214958 25354 hybrid_clock.cc:648] HybridClock initialized: now 1790481801214941 us; error 27 us; skew 500 ppm
Sep 27 04:03:21 dist-test-slave-zqrn krb5kdc[24200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481801, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:21.217540 25354 init.cc:377] Logged in from keytab as oryx/127.17.184.195@KRBTEST.COM (short username oryx)
I20260927 04:03:21.218034 25354 webserver.cc:460] Webserver started at http://127.17.184.195:40753/ using document root <none> and password file <none>
I20260927 04:03:21.218276 25354 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:21.218358 25354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:21.219811 25354 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:21.220460 25369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:21.220633 25354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:21.220736 25354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "5d155427f4504ceca4f5a00791005a3b"
format_stamp: "Formatted at 2026-09-27 04:03:20 on dist-test-slave-zqrn"
server_key: "ac05361d97b99d90c105b6f791507762"
server_key_iv: "be193c294ae3d7a091b3099fd0c5989c"
server_key_version: "encryptionkey@0"
I20260927 04:03:21.221076 25354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:21.239034 25354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:21.241444 25354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:21.241611 25354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:21.241858 25354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:21.242182 25354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:21.242247 25354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:21.242312 25354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:21.242342 25354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:21.248960 25354 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:34693
I20260927 04:03:21.249083 25482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:34693 every 8 connection(s)
I20260927 04:03:21.249408 25354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:21.255905 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 25354
Sep 27 04:03:21 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481801, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
Sep 27 04:03:21 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481801, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.252@KRBTEST.COM
Sep 27 04:03:21 dist-test-slave-zqrn krb5kdc[24200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481801, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.253@KRBTEST.COM
I20260927 04:03:21.263080 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.250241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:57259)
0927 04:03:21.250409 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.250453 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.251077 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.251327 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.251331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.251744 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:21.251952 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.251956 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.252672 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.252673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.253680 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.253682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.254156 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.254157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.254378 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.255217 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:21.255228 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:21.257949 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:21.260759 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.260766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.260768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.261571 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.261796 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.261797 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.261797 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.261937 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.262259 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:21.262267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:21.262445 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.262577 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.262782 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":8,"thread_start_us":115,"threads_started":1}
I20260927 04:03:21.263322 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.250240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.17.184.195:53787)
0927 04:03:21.250690 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.250699 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.251102 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.251728 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.251728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.251885 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:21.251953 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.251960 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.252920 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.252921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.253850 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.253852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.254345 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.254346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.254398 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.255217 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:21.255228 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:21.258302 (+ 3074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:21.260990 (+ 2688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.260991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.260991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.261680 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.261890 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.261891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.261891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.261938 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.262259 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:21.262267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:21.262445 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.262577 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.263192 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":503,"thread_start_us":89,"threads_started":1}
I20260927 04:03:21.263800 24284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.250260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:57259 (local address 127.17.184.254:34757)
0927 04:03:21.250306 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.250309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.250564 (+ 255us) server_negotiation.cc:407] Connection header received
0927 04:03:21.251181 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.251182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.251195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.251259 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.252044 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.252565 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.253782 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.254073 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.254141 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.258753 (+ 4612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.258754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.258755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.258762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.260651 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.261658 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.261659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.261660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.261724 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.262006 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.262007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.262008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.262155 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.262231 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.263616 (+ 1385us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.263704 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:03:21.263916 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.250244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:53787 (local address 127.17.184.252:36099)
0927 04:03:21.250302 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.250304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.250748 (+ 444us) server_negotiation.cc:407] Connection header received
0927 04:03:21.251208 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.251210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.251222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.251253 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.252059 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.252820 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.253963 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.254218 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.254296 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.258394 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.258396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.258396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.258405 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.260909 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.261767 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.261768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.261768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.261825 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.262006 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.262007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.262007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.262155 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.262230 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.263729 (+ 1499us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.263809 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 04:03:21.264290 25484 heartbeater.cc:347] Connected to a master server at 127.17.184.254:34757
I20260927 04:03:21.264394 25484 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:21.264621 25484 heartbeater.cc:511] Master 127.17.184.254:34757 requested a full tablet report, sending...
I20260927 04:03:21.265030 24245 ts_manager.cc:194] Re-registered known tserver with Master: 5d155427f4504ceca4f5a00791005a3b (127.17.184.195:34693)
I20260927 04:03:21.265513 24245 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:57259
I20260927 04:03:21.265592 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.256996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.17.184.252:36099)
0927 04:03:21.257255 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.257257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.257472 (+ 215us) server_negotiation.cc:407] Connection header received
0927 04:03:21.257584 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.257585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.257598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.257875 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.258460 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.259257 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.260399 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.260922 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.261236 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.262414 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.262415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.262415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.262422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.264175 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.264564 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.264565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.264565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.264622 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.264848 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.264849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.264849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.264984 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.265367 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.265413 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.265494 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":49,"threads_started":1}
I20260927 04:03:21.266520 25483 heartbeater.cc:347] Connected to a master server at 127.17.184.252:36099
I20260927 04:03:21.266566 25483 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:21.266645 25483 heartbeater.cc:511] Master 127.17.184.252:36099 requested a full tablet report, sending...
I20260927 04:03:21.266943 25023 ts_manager.cc:194] Registered new tserver with Master: 5d155427f4504ceca4f5a00791005a3b (127.17.184.195:34693)
I20260927 04:03:21.267727 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.257031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.17.184.254:34757)
0927 04:03:21.257319 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.257321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.257329 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:21.257386 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.257388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.257404 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.257453 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.258458 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.259250 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.260416 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.260732 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.260777 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.262422 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.262423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.262423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.262429 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.266369 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.266766 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.266767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.266767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.266826 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.267064 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.267065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.267065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.267196 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.267274 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.267517 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.267611 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":71,"threads_started":1}
I20260927 04:03:21.267961 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.250629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:37201 (local address 127.17.184.253:38373)
0927 04:03:21.250686 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.250688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.250870 (+ 182us) server_negotiation.cc:407] Connection header received
0927 04:03:21.251202 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.251203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.251216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.251250 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.252060 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.252825 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.253930 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.254221 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.254283 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.264622 (+ 10339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.264624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.264624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.264632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.266330 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.267111 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.267112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.267112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.267167 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.267383 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.267384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.267384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.267526 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.267587 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.267783 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.267866 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:03:21.268364 25488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.250557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.17.184.195:37201)
0927 04:03:21.250798 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.250806 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.251102 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.251346 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.251346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.251787 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:21.251953 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.251959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.252920 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.252921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.253853 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.253855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.254294 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.254295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.254382 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.255246 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:21.255251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:21.264508 (+ 9257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:21.266417 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.266418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.266418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.266708 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.267237 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:21.267238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:21.267238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:21.267306 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:21.267624 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:21.267625 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:21.267667 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.268071 (+ 404us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.268249 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":49,"thread_start_us":62,"threads_started":1}
I20260927 04:03:21.268735 25485 heartbeater.cc:347] Connected to a master server at 127.17.184.253:38373
I20260927 04:03:21.268787 25485 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:21.268836 25485 heartbeater.cc:511] Master 127.17.184.253:38373 requested a full tablet report, sending...
I20260927 04:03:21.269134 24843 ts_manager.cc:194] Registered new tserver with Master: 5d155427f4504ceca4f5a00791005a3b (127.17.184.195:34693)
I20260927 04:03:21.270005 24245 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:21.271673 24245 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:21.272759 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.257318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.17.184.253:38373)
0927 04:03:21.257604 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.257606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.257694 (+ 88us) server_negotiation.cc:407] Connection header received
0927 04:03:21.257795 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.257797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.257808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.257844 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.258585 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.263659 (+ 5074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.267554 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.267833 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.267905 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.270029 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.270031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.270031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.270038 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.271497 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.271888 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.271889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.271890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.271945 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.272150 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.272151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.272151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.272270 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.272387 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.272583 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.272667 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":48,"threads_started":1}
I20260927 04:03:21.279917 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.275843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.17.184.195:34693)
0927 04:03:21.276302 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.276307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.276319 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:03:21.276361 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.276363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.276416 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.276504 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:21.276617 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.277731 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.279031 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.279550 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.279655 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.279718 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.279799 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.279837 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":111,"threads_started":1}
I20260927 04:03:21.279904 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.275682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34693 (local address 127.0.0.1:45572)
0927 04:03:21.275725 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.275735 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.275794 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.276499 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.276500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.276501 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:21.276550 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.276551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.277824 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.277825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.278969 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.278971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.279654 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.279655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.279721 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.279725 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.279782 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.279823 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":26}
I20260927 04:03:21.280148 25504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.275830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33249 (local address 127.0.0.1:38342)
0927 04:03:21.275979 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.275990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.276034 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.276801 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.276802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.276804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:21.276847 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.276848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.277708 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.277709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.278937 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.278941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.279888 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.279888 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.279956 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.279960 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.279992 (+ 32us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.280037 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":8,"thread_start_us":93,"threads_started":1}
I20260927 04:03:21.280230 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.276035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:45969 (local address 127.0.0.1:41982)
0927 04:03:21.276157 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.276166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.276201 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.276647 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.276648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.276650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:21.276713 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.276714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.278040 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.278041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.279021 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.279025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.279886 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.279887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.279959 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.279961 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.279992 (+ 31us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.280037 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260927 04:03:21.280441 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.276229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41982 (local address 127.17.184.194:45969)
0927 04:03:21.276391 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.276394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.276402 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:21.276452 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.276455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.276470 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.276652 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:21.276937 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.277956 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.279091 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.279781 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.280106 (+ 325us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.280168 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.280194 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.280359 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260927 04:03:21.280534 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.275921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.17.184.193:33249)
0927 04:03:21.276627 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.276631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.276641 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:21.276708 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.276710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.276727 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.276802 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:21.276916 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.277595 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.279046 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.279785 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.280312 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.280374 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.280401 (+ 27us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.280445 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":72,"threads_started":1}
I20260927 04:03:21.281169 25144 tablet_service.cc:1467] Processing CreateTablet for tablet 90ac3981262445c88cf2030245a4e1fc (DEFAULT_TABLE table=test-table [id=8761dedbedd94519a94ea3aa0aa7dc6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:21.281179 25281 tablet_service.cc:1467] Processing CreateTablet for tablet 90ac3981262445c88cf2030245a4e1fc (DEFAULT_TABLE table=test-table [id=8761dedbedd94519a94ea3aa0aa7dc6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:21.281545 25281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ac3981262445c88cf2030245a4e1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:21.281545 25144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ac3981262445c88cf2030245a4e1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:21.283607 25417 tablet_service.cc:1467] Processing CreateTablet for tablet 90ac3981262445c88cf2030245a4e1fc (DEFAULT_TABLE table=test-table [id=8761dedbedd94519a94ea3aa0aa7dc6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:21.283950 25417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90ac3981262445c88cf2030245a4e1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:21.285272 25509 tablet_bootstrap.cc:492] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Bootstrap starting.
I20260927 04:03:21.285458 25510 tablet_bootstrap.cc:492] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Bootstrap starting.
I20260927 04:03:21.286062 25510 tablet_bootstrap.cc:654] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:21.286178 25509 tablet_bootstrap.cc:654] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:21.286392 25510 log.cc:824] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:21.286511 25509 log.cc:824] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:21.287166 25510 tablet_bootstrap.cc:492] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: No bootstrap required, opened a new log
I20260927 04:03:21.287248 25510 ts_tablet_manager.cc:1397] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:21.287365 25509 tablet_bootstrap.cc:492] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: No bootstrap required, opened a new log
I20260927 04:03:21.287441 25509 ts_tablet_manager.cc:1397] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:21.287942 25513 tablet_bootstrap.cc:492] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Bootstrap starting.
I20260927 04:03:21.288591 25513 tablet_bootstrap.cc:654] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:21.288992 25513 log.cc:824] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:21.289038 25509 raft_consensus.cc:348] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.289209 25509 raft_consensus.cc:374] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:21.289258 25509 raft_consensus.cc:729] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84555cae7414ffc947a877ac4315649, State: Initialized, Role: FOLLOWER
I20260927 04:03:21.289390 25509 consensus_queue.cc:260] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.289723 25348 heartbeater.cc:503] Master 127.17.184.254:34757 was elected leader, sending a full tablet report...
I20260927 04:03:21.289793 25509 ts_tablet_manager.cc:1428] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:21.289880 25510 raft_consensus.cc:348] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.290149 25510 raft_consensus.cc:374] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:21.290237 25510 raft_consensus.cc:729] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d807011b38409097e30e028fb20cbb, State: Initialized, Role: FOLLOWER
I20260927 04:03:21.290390 25510 consensus_queue.cc:260] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.290704 25510 ts_tablet_manager.cc:1428] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 04:03:21.290937 25211 heartbeater.cc:503] Master 127.17.184.254:34757 was elected leader, sending a full tablet report...
I20260927 04:03:21.292232 25513 tablet_bootstrap.cc:492] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: No bootstrap required, opened a new log
I20260927 04:03:21.292348 25513 ts_tablet_manager.cc:1397] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 04:03:21.294708 25513 raft_consensus.cc:348] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.294920 25513 raft_consensus.cc:374] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:21.294974 25513 raft_consensus.cc:729] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d155427f4504ceca4f5a00791005a3b, State: Initialized, Role: FOLLOWER
I20260927 04:03:21.295112 25513 consensus_queue.cc:260] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.295401 25513 ts_tablet_manager.cc:1428] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:03:21.295475 25484 heartbeater.cc:503] Master 127.17.184.254:34757 was elected leader, sending a full tablet report...
I20260927 04:03:21.331611 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.329072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60142 (local address 127.17.184.252:36099)
0927 04:03:21.329153 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.329155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.329170 (+ 15us) server_negotiation.cc:407] Connection header received
0927 04:03:21.329200 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.329201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.329222 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.329318 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:21.329479 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.329971 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.330873 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.331333 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.331395 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.331451 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.331524 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.331553 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:03:21.331614 25507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.328978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.0.0.1:60142)
0927 04:03:21.329033 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.329047 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.329097 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.329345 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.329346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.329347 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:21.329411 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.329412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.330048 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.330049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.330790 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.330792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.331414 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.331414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.331464 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.331467 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.331514 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.331552 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 04:03:21.332096 25041 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260927 04:03:21.333029 25070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ba21c2b6e0047109248e6ca90d28c85. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:21.333138 25070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: This master's current role is: LEARNER
W20260927 04:03:21.356464 25350 tablet.cc:2369] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:03:21.457257 25213 tablet.cc:2369] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:03:21.500236 25486 tablet.cc:2369] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:21.527599 25516 raft_consensus.cc:482] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:21.527781 25516 raft_consensus.cc:504] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.528203 25516 leader_election.cc:288] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d155427f4504ceca4f5a00791005a3b (127.17.184.195:34693), e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969)
I20260927 04:03:21.531772 25522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.528425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:45969 (local address 127.17.184.193:39981)
0927 04:03:21.528556 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.528568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.528639 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.528874 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.528875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.528878 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:21.528963 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.528966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.529874 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.529876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.530714 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.530717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.531490 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.531491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.531557 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.531562 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.531628 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.531685 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260927 04:03:21.531772 25508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.528518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:39981 (local address 127.17.184.194:45969)
0927 04:03:21.528573 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.528576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.528637 (+ 61us) server_negotiation.cc:407] Connection header received
0927 04:03:21.528708 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.528709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.528720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.528825 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:21.529049 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.529736 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.530819 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.531402 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.531489 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.531498 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.531659 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.531697 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 04:03:21.532282 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ac3981262445c88cf2030245a4e1fc" candidate_uuid: "60d807011b38409097e30e028fb20cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84555cae7414ffc947a877ac4315649" is_pre_election: true
I20260927 04:03:21.532497 25301 raft_consensus.cc:2454] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d807011b38409097e30e028fb20cbb in term 0.
I20260927 04:03:21.532598 25505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.528498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:54511 (local address 127.17.184.195:34693)
0927 04:03:21.528554 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.528556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.528644 (+ 88us) server_negotiation.cc:407] Connection header received
0927 04:03:21.528733 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.528734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.528744 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.528857 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:21.529865 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.530605 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.531679 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.532221 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.532335 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.532343 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.532495 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.532534 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 04:03:21.532614 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.528444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34693 (local address 127.17.184.193:54511)
0927 04:03:21.528557 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:21.528567 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:21.528658 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:21.528885 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:21.528886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:21.528887 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:21.528963 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.528965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.530726 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.530727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.531564 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:21.531566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.532335 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.532336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:21.532395 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.532398 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:21.532483 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 04:03:21.532530 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":19,"thread_start_us":51,"threads_started":1}
I20260927 04:03:21.532801 25097 leader_election.cc:302] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d807011b38409097e30e028fb20cbb, e84555cae7414ffc947a877ac4315649; no voters:
I20260927 04:03:21.532922 25516 raft_consensus.cc:2781] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:21.532948 25437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ac3981262445c88cf2030245a4e1fc" candidate_uuid: "60d807011b38409097e30e028fb20cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d155427f4504ceca4f5a00791005a3b" is_pre_election: true
I20260927 04:03:21.533002 25516 raft_consensus.cc:482] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:21.533030 25516 raft_consensus.cc:3037] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:21.533088 25437 raft_consensus.cc:2454] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d807011b38409097e30e028fb20cbb in term 0.
I20260927 04:03:21.533846 25516 raft_consensus.cc:504] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.533993 25516 leader_election.cc:288] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [CANDIDATE]: Term 1 election: Requested vote from peers 5d155427f4504ceca4f5a00791005a3b (127.17.184.195:34693), e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969)
I20260927 04:03:21.534163 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ac3981262445c88cf2030245a4e1fc" candidate_uuid: "60d807011b38409097e30e028fb20cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84555cae7414ffc947a877ac4315649"
I20260927 04:03:21.534163 25437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90ac3981262445c88cf2030245a4e1fc" candidate_uuid: "60d807011b38409097e30e028fb20cbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d155427f4504ceca4f5a00791005a3b"
I20260927 04:03:21.534238 25301 raft_consensus.cc:3037] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:21.534238 25437 raft_consensus.cc:3037] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:21.535188 25437 raft_consensus.cc:2454] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d807011b38409097e30e028fb20cbb in term 1.
I20260927 04:03:21.535188 25301 raft_consensus.cc:2454] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d807011b38409097e30e028fb20cbb in term 1.
I20260927 04:03:21.535522 25097 leader_election.cc:302] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d807011b38409097e30e028fb20cbb, e84555cae7414ffc947a877ac4315649; no voters:
I20260927 04:03:21.535727 25516 raft_consensus.cc:2781] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:21.535880 25516 raft_consensus.cc:686] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 1 LEADER]: Becoming Leader. State: Replica: 60d807011b38409097e30e028fb20cbb, State: Running, Role: LEADER
I20260927 04:03:21.535977 25516 consensus_queue.cc:237] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } }
I20260927 04:03:21.536813 24245 catalog_manager.cc:5495] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d807011b38409097e30e028fb20cbb (127.17.184.193). New cstate: current_term: 1 leader_uuid: "60d807011b38409097e30e028fb20cbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:21.537458 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: NON_VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: true } } }
I20260927 04:03:21.537554 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:21.538251 25525 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: attempting to promote NON_VOTER ae095317e0aa47fb85ea3292e86dfe17 to VOTER
I20260927 04:03:21.538399 25525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } }
I20260927 04:03:21.538856 24853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Refusing update from remote peer 4ba21c2b6e0047109248e6ca90d28c85: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 04:03:21.538880 25041 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 LEARNER]: Refusing update from remote peer 4ba21c2b6e0047109248e6ca90d28c85: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260927 04:03:21.539114 25525 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260927 04:03:21.539249 24994 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260927 04:03:21.540010 24804 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:21.540150 24853 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:21.540148 25041 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:21.540898 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:21.540977 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:21.541039 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:21.541088 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:21.541136 24804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:21.541184 24804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:21.541429 25070 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:21.541543 25070 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:03:21.541800 25527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ba21c2b6e0047109248e6ca90d28c85" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:21.541939 25527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:03:21.571591 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.568787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.17.184.193:33249)
0927 04:03:21.568857 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.568859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.568869 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:21.568910 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.568912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.568942 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.569032 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:21.569207 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.569779 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.570681 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.570931 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.571028 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.571219 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:21.571342 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:21.571450 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.571470 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.571494 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:03:21.573668 25437 raft_consensus.cc:1261] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 1 FOLLOWER]: Refusing update from remote peer 60d807011b38409097e30e028fb20cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:21.573697 25301 raft_consensus.cc:1261] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 1 FOLLOWER]: Refusing update from remote peer 60d807011b38409097e30e028fb20cbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:21.574093 25516 consensus_queue.cc:1035] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:21.574250 25523 consensus_queue.cc:1035] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:21.581545 24245 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:21.581655 24245 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:21.583277 24245 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ba21c2b6e0047109248e6ca90d28c85: Sending DeleteTablet for 3 replicas of tablet 90ac3981262445c88cf2030245a4e1fc
I20260927 04:03:21.583585 25417 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ac3981262445c88cf2030245a4e1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:21 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:45572
I20260927 04:03:21.583752 25144 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ac3981262445c88cf2030245a4e1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:21 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:38342
I20260927 04:03:21.583778 25281 tablet_service.cc:1514] Processing DeleteTablet for tablet 90ac3981262445c88cf2030245a4e1fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:21 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41982
I20260927 04:03:21.584007 25538 tablet_replica.cc:330] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: stopping tablet replica
I20260927 04:03:21.584118 25538 raft_consensus.cc:2229] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:21.584259 25538 raft_consensus.cc:2258] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:21.584775 25538 ts_tablet_manager.cc:1905] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:21.584983 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 24213
I20260927 04:03:21.586875 25538 ts_tablet_manager.cc:1918] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:21.586998 25538 log.cc:1196] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/90ac3981262445c88cf2030245a4e1fc
I20260927 04:03:21.587291 25538 ts_tablet_manager.cc:1939] T 90ac3981262445c88cf2030245a4e1fc P 5d155427f4504ceca4f5a00791005a3b: Deleting consensus metadata
I20260927 04:03:21.592753 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.584025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.17.184.252:36099)
0927 04:03:21.584068 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.584071 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.584196 (+ 125us) server_negotiation.cc:407] Connection header received
0927 04:03:21.584301 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.584302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.584314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.584347 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.585278 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.586096 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.587559 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.587833 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.587871 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.589709 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.589710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.589710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.589718 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.591465 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.591831 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.591832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.591833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.591893 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.592126 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.592127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.592127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.592293 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.592420 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.592538 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.592624 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:03:21.593252 25370 connection.cc:535] server connection from 127.0.0.1:45572 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 04:03:21.594091 25539 tablet_replica.cc:330] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: stopping tablet replica
I20260927 04:03:21.594252 25539 raft_consensus.cc:2229] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:21.594399 25539 raft_consensus.cc:2258] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:21.594604 25540 tablet_replica.cc:330] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: stopping tablet replica
I20260927 04:03:21.594753 25540 raft_consensus.cc:2229] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:21.594971 25540 raft_consensus.cc:2258] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:21.595072 25539 ts_tablet_manager.cc:1905] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:21.595510 25540 ts_tablet_manager.cc:1905] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:21.597131 25540 ts_tablet_manager.cc:1918] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:21.597150 25539 ts_tablet_manager.cc:1918] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:21.597218 25539 log.cc:1196] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/90ac3981262445c88cf2030245a4e1fc
I20260927 04:03:21.597298 25540 log.cc:1196] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/90ac3981262445c88cf2030245a4e1fc
I20260927 04:03:21.597499 25539 ts_tablet_manager.cc:1939] T 90ac3981262445c88cf2030245a4e1fc P e84555cae7414ffc947a877ac4315649: Deleting consensus metadata
I20260927 04:03:21.597680 25540 ts_tablet_manager.cc:1939] T 90ac3981262445c88cf2030245a4e1fc P 60d807011b38409097e30e028fb20cbb: Deleting consensus metadata
W20260927 04:03:21.598022 25097 connection.cc:441] server connection from 127.0.0.1:38342 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38342 (request call id 1) could send its response
W20260927 04:03:21.598229 25236 connection.cc:441] server connection from 127.0.0.1:41982 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41982 (request call id 1) could send its response
I20260927 04:03:21.604192 25498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.595053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54198 (local address 127.17.184.253:38373)
0927 04:03:21.595104 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.595106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.595621 (+ 515us) server_negotiation.cc:407] Connection header received
0927 04:03:21.595728 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.595729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.595746 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.595780 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.596657 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.597518 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.598749 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.599171 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.599246 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.600786 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.600788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.600788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.600795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.602635 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.603216 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.603217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.603217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.603283 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.603496 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.603497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.603498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.603656 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.603790 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.603910 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.604098 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:03:21.606269 25494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:21.597836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60168 (local address 127.17.184.252:36099)
0927 04:03:21.597884 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:21.597886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:21.598959 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 04:03:21.599059 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:21.599060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:21.599071 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:21.599102 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:21.599894 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.600409 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.601403 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:21.601751 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:21.601818 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:21.603391 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:21.603393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:21.603393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:21.603400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:21.605186 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.605514 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.605515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.605516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.605575 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:21.605773 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:21.605774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:21.605774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:21.605903 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:21.605998 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:21.606103 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 04:03:21.606176 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:03:22.579363 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:22.579022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:37875)
0927 04:03:22.579200 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:22.579300 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
I20260927 04:03:22.579408 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:22.579126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:45817)
0927 04:03:22.579280 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:22.579353 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
W20260927 04:03:22.579609 25211 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:34757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:22.579777 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:22.579554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:40599)
0927 04:03:22.579665 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:22.579729 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":49,"threads_started":1}
W20260927 04:03:22.579783 25484 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:34757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
W20260927 04:03:22.579932 25348 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:34757 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:23.083561 25552 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ba21c2b6e0047109248e6ca90d28c85)
I20260927 04:03:23.083757 25552 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } }
I20260927 04:03:23.084012 25552 leader_election.cc:288] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757), 0d6696016cb848b08265756e11260656 (127.17.184.253:38373)
I20260927 04:03:23.084288 25553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ba21c2b6e0047109248e6ca90d28c85)
I20260927 04:03:23.084446 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.084194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:40220)
0927 04:03:23.084342 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.084385 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260927 04:03:23.084424 25553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } }
W20260927 04:03:23.084599 25016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:23.084685 25553 leader_election.cc:288] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757), ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252:36099)
I20260927 04:03:23.085181 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.084856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:40232)
0927 04:03:23.085072 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.085136 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":61,"threads_started":1}
I20260927 04:03:23.085348 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.085253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:40248)
0927 04:03:23.085294 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.085319 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 04:03:23.085398 24828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
W20260927 04:03:23.085430 25016 leader_election.cc:334] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757): Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:23.085897 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.085777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:40262)
0927 04:03:23.085836 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.085870 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 04:03:23.085978 24828 leader_election.cc:334] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757): Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:23.088349 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.084197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:38373 (local address 127.0.0.1:54212)
0927 04:03:23.084374 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.084385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:23.084450 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:23.084740 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:23.084741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:23.084744 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:23.084854 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:23.084857 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:23.086053 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:23.086054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:23.087132 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:23.087134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:23.088034 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:23.088035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:23.088108 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:23.088111 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:23.088205 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 04:03:23.088252 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":64,"threads_started":1}
I20260927 04:03:23.088487 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.084835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:36099 (local address 127.0.0.1:60182)
0927 04:03:23.084992 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.085002 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:23.085060 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:23.085334 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:23.085335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:23.085337 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:23.085386 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:23.085388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:23.086381 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:23.086382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:23.087392 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:23.087395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:23.088205 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:23.088206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:23.088280 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:23.088284 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:23.088345 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 04:03:23.088396 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260927 04:03:23.088500 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.085009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.17.184.252:36099)
0927 04:03:23.085174 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:23.085176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:23.085184 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:23.085210 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:23.085210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:23.085221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:23.085289 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:23.085470 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:23.086252 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:23.087500 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:23.088109 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:23.088209 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:23.088217 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:23.088362 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 04:03:23.088412 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260927 04:03:23.088361 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.084392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.17.184.253:38373)
0927 04:03:23.084577 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:23.084579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:23.084587 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:23.084618 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:23.084619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:23.084631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:23.084934 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:23.084990 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:23.085928 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:23.087237 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:23.087862 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:23.087949 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:23.087959 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:23.088219 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 04:03:23.088275 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":45,"threads_started":1}
I20260927 04:03:23.088763 25041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d6696016cb848b08265756e11260656" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ae095317e0aa47fb85ea3292e86dfe17" is_pre_election: true
I20260927 04:03:23.088879 25041 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d6696016cb848b08265756e11260656 in term 1.
I20260927 04:03:23.088853 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae095317e0aa47fb85ea3292e86dfe17" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d6696016cb848b08265756e11260656" is_pre_election: true
I20260927 04:03:23.088970 24853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae095317e0aa47fb85ea3292e86dfe17 in term 1.
I20260927 04:03:23.089066 24827 leader_election.cc:302] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d6696016cb848b08265756e11260656, ae095317e0aa47fb85ea3292e86dfe17; no voters: 4ba21c2b6e0047109248e6ca90d28c85
I20260927 04:03:23.089174 25553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 04:03:23.089221 25553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4ba21c2b6e0047109248e6ca90d28c85)
I20260927 04:03:23.089254 25553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:03:23.089274 25018 leader_election.cc:302] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d6696016cb848b08265756e11260656, ae095317e0aa47fb85ea3292e86dfe17; no voters: 4ba21c2b6e0047109248e6ca90d28c85
I20260927 04:03:23.089367 25552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 04:03:23.089442 25552 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4ba21c2b6e0047109248e6ca90d28c85)
I20260927 04:03:23.089461 25552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:03:23.090104 25553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } }
I20260927 04:03:23.090245 25553 leader_election.cc:288] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [CANDIDATE]: Term 2 election: Requested vote from peers 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757), ae095317e0aa47fb85ea3292e86dfe17 (127.17.184.252:36099)
I20260927 04:03:23.090250 25552 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } }
I20260927 04:03:23.090373 25041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d6696016cb848b08265756e11260656" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ae095317e0aa47fb85ea3292e86dfe17"
I20260927 04:03:23.090466 25041 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d6696016cb848b08265756e11260656 in current term 2: Already voted for candidate ae095317e0aa47fb85ea3292e86dfe17 in this term.
I20260927 04:03:23.090404 25552 leader_election.cc:288] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 2 election: Requested vote from peers 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757), 0d6696016cb848b08265756e11260656 (127.17.184.253:38373)
I20260927 04:03:23.090516 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae095317e0aa47fb85ea3292e86dfe17" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d6696016cb848b08265756e11260656"
I20260927 04:03:23.090600 24853 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ae095317e0aa47fb85ea3292e86dfe17 in current term 2: Already voted for candidate 0d6696016cb848b08265756e11260656 in this term.
I20260927 04:03:23.090656 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.090528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:40278)
0927 04:03:23.090589 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.090620 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 04:03:23.090975 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.090344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:40268)
0927 04:03:23.090840 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.090936 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 04:03:23.090984 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.090874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:40280)
0927 04:03:23.090936 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.090960 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 04:03:23.091097 25016 leader_election.cc:334] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757): Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:23.091145 25016 leader_election.cc:302] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae095317e0aa47fb85ea3292e86dfe17; no voters: 0d6696016cb848b08265756e11260656, 4ba21c2b6e0047109248e6ca90d28c85
I20260927 04:03:23.091307 25552 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 04:03:23.091554 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.091454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:40286)
0927 04:03:23.091508 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.091534 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 04:03:23.091634 24828 leader_election.cc:334] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757): Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:23.091707 24828 leader_election.cc:302] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d6696016cb848b08265756e11260656; no voters: 4ba21c2b6e0047109248e6ca90d28c85, ae095317e0aa47fb85ea3292e86dfe17
I20260927 04:03:23.091827 25553 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 04:03:23.580202 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.579954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:50929)
0927 04:03:23.580101 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.580144 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260927 04:03:23.580312 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.580091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:58115)
0927 04:03:23.580219 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.580261 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260927 04:03:23.580379 25564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:23.580171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:59051)
0927 04:03:23.580302 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:23.580334 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":48,"threads_started":1}
I20260927 04:03:24.580895 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.580613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:48107)
0927 04:03:24.580788 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.580835 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":119,"threads_started":1}
I20260927 04:03:24.581097 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.580709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:33975)
0927 04:03:24.581020 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.581057 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":107,"threads_started":1}
I20260927 04:03:24.581079 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.580647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:52183)
0927 04:03:24.580990 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.581029 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":42,"threads_started":1}
I20260927 04:03:24.718113 25568 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:24.718268 25568 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } }
I20260927 04:03:24.718530 25568 leader_election.cc:288] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757), 0d6696016cb848b08265756e11260656 (127.17.184.253:38373)
I20260927 04:03:24.718967 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae095317e0aa47fb85ea3292e86dfe17" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d6696016cb848b08265756e11260656" is_pre_election: true
I20260927 04:03:24.719074 24853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae095317e0aa47fb85ea3292e86dfe17 in term 2.
I20260927 04:03:24.719120 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.718766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:35880)
0927 04:03:24.718997 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.719045 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260927 04:03:24.719302 25018 leader_election.cc:302] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d6696016cb848b08265756e11260656, ae095317e0aa47fb85ea3292e86dfe17; no voters:
I20260927 04:03:24.719403 25568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 04:03:24.719455 25568 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:24.719477 25568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 2 FOLLOWER]: Advancing to term 3
I20260927 04:03:24.719684 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.719570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:35892)
0927 04:03:24.719627 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.719659 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 04:03:24.719798 25016 leader_election.cc:334] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757): Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:24.720366 25568 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } }
I20260927 04:03:24.720507 25568 leader_election.cc:288] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 3 election: Requested vote from peers 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757), 0d6696016cb848b08265756e11260656 (127.17.184.253:38373)
I20260927 04:03:24.720700 24853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae095317e0aa47fb85ea3292e86dfe17" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0d6696016cb848b08265756e11260656"
I20260927 04:03:24.720736 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.720613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:35904)
0927 04:03:24.720671 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.720700 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 04:03:24.720779 24853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 2 FOLLOWER]: Advancing to term 3
I20260927 04:03:24.721136 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.721052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:35920)
0927 04:03:24.721095 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.721120 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:03:24.721238 25016 leader_election.cc:334] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757): Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
I20260927 04:03:24.721597 24853 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae095317e0aa47fb85ea3292e86dfe17 in term 3.
I20260927 04:03:24.721760 25018 leader_election.cc:302] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d6696016cb848b08265756e11260656, ae095317e0aa47fb85ea3292e86dfe17; no voters: 4ba21c2b6e0047109248e6ca90d28c85
I20260927 04:03:24.721855 25568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 04:03:24.721961 25568 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [term 3 LEADER]: Becoming Leader. State: Replica: ae095317e0aa47fb85ea3292e86dfe17, State: Running, Role: LEADER
I20260927 04:03:24.722043 25568 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } }
I20260927 04:03:24.722415 25571 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae095317e0aa47fb85ea3292e86dfe17. Latest consensus state: current_term: 3 leader_uuid: "ae095317e0aa47fb85ea3292e86dfe17" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:24.722522 25571 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:24.722692 25573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:24.723169 25573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:24.723230 25573 catalog_manager.cc:1259] Loaded cluster ID: c86a0064840946bcb8825eaa90b1706f
I20260927 04:03:24.723246 25573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:24.723387 25573 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:24.723480 25573 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Loaded TSK: 0
I20260927 04:03:24.723624 25573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:03:24.779035 25031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:24.780671 25031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:24.781781 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.781635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:35938)
0927 04:03:24.781690 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.781746 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:03:24.781808 24853 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [term 3 FOLLOWER]: Refusing update from remote peer ae095317e0aa47fb85ea3292e86dfe17: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 04:03:24.782047 25568 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 04:03:24.782210 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.782104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.0.0.1:35946)
0927 04:03:24.782158 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.782184 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 04:03:24.782342 25016 consensus_peers.cc:487] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 -> Peer 4ba21c2b6e0047109248e6ca90d28c85 (127.17.184.254:34757): Couldn't send request to peer 4ba21c2b6e0047109248e6ca90d28c85. Status: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 04:03:24.782763 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae095317e0aa47fb85ea3292e86dfe17. Latest consensus state: current_term: 3 leader_uuid: "ae095317e0aa47fb85ea3292e86dfe17" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:24.782877 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:03:24.783440 25568 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ae095317e0aa47fb85ea3292e86dfe17" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:24.783535 25568 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:24.783595 25576 mvcc.cc:205] Tried to move back new op lower bound from 7333813472383877120 to 7333813472142028800. Current Snapshot: MvccSnapshot[applied={T|T < 7333813472383877120}]
I20260927 04:03:24.783782 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ae095317e0aa47fb85ea3292e86dfe17" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "4ba21c2b6e0047109248e6ca90d28c85" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 34757 } } peers { permanent_uuid: "0d6696016cb848b08265756e11260656" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 38373 } attrs { promote: false } } peers { permanent_uuid: "ae095317e0aa47fb85ea3292e86dfe17" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 36099 } attrs { promote: false } } }
I20260927 04:03:24.783872 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d6696016cb848b08265756e11260656 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:03:24.789222 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.785414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:45969 (local address 127.0.0.1:40286)
0927 04:03:24.785449 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.785461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:24.786074 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:24.786321 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:24.786322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:24.786323 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:24.786356 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.786357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.787240 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.787242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.788168 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.788171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.788930 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.788931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.788997 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.789002 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:03:24.789058 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 04:03:24.789104 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 04:03:24.789215 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.785512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.17.184.194:45969)
0927 04:03:24.786183 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:24.786185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:24.786193 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:24.786220 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:24.786221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:24.786233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:24.786318 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:24.786408 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.787135 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.788260 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.788837 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.788924 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.788934 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:24.789089 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 04:03:24.789134 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":87,"threads_started":1}
I20260927 04:03:24.789354 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.785781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44234 (local address 127.17.184.193:33249)
0927 04:03:24.785886 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:24.785888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:24.785897 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:24.785929 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:24.785930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:24.785942 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:24.786029 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:24.786236 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.787086 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.788360 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.788943 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.789014 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.789020 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:24.789209 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 04:03:24.789266 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260927 04:03:24.789547 25582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.785517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33249 (local address 127.0.0.1:44234)
0927 04:03:24.785684 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.785695 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:24.785748 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:24.786037 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:24.786038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:24.786040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:24.786092 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.786094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.787174 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.787174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.788276 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.788279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.789037 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.789038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.789094 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.789097 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:24.789143 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 04:03:24.789175 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":7,"thread_start_us":87,"threads_started":1}
I20260927 04:03:24.789758 25281 tablet_service.cc:1467] Processing CreateTablet for tablet 8352d7c385b848b0bc6818838afc3530 (DEFAULT_TABLE table=test-table [id=be11b014e4034940835faa2db0c81085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:24.789853 25144 tablet_service.cc:1467] Processing CreateTablet for tablet 8352d7c385b848b0bc6818838afc3530 (DEFAULT_TABLE table=test-table [id=be11b014e4034940835faa2db0c81085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:24.789908 25281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8352d7c385b848b0bc6818838afc3530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:24.789958 25144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8352d7c385b848b0bc6818838afc3530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:24.790113 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.785658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.17.184.195:34693)
0927 04:03:24.785805 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:24.785807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:24.785886 (+ 79us) server_negotiation.cc:407] Connection header received
0927 04:03:24.785946 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:24.785947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:24.785959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:24.786026 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:24.786205 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.786942 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.789082 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.789681 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.789774 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.789782 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:24.789934 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 04:03:24.790021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260927 04:03:24.790364 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.785553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34693 (local address 127.0.0.1:56102)
0927 04:03:24.785832 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.785842 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:24.785892 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:24.786084 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:24.786085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:24.786087 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:24.786123 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.786125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.787014 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.787015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.788276 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.788279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.789774 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.789775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.789839 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.789844 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:03:24.789913 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 04:03:24.790291 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":2,"thread_start_us":72,"threads_started":1}
I20260927 04:03:24.790649 25417 tablet_service.cc:1467] Processing CreateTablet for tablet 8352d7c385b848b0bc6818838afc3530 (DEFAULT_TABLE table=test-table [id=be11b014e4034940835faa2db0c81085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:24.790810 25417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8352d7c385b848b0bc6818838afc3530. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:24.791484 25586 tablet_bootstrap.cc:492] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: Bootstrap starting.
I20260927 04:03:24.791563 25587 tablet_bootstrap.cc:492] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb: Bootstrap starting.
I20260927 04:03:24.791980 25586 tablet_bootstrap.cc:654] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:24.792013 25587 tablet_bootstrap.cc:654] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:24.792569 25587 tablet_bootstrap.cc:492] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb: No bootstrap required, opened a new log
I20260927 04:03:24.792611 25587 ts_tablet_manager.cc:1397] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:24.792709 25587 raft_consensus.cc:348] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.792766 25587 raft_consensus.cc:374] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:24.792780 25587 raft_consensus.cc:729] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d807011b38409097e30e028fb20cbb, State: Initialized, Role: FOLLOWER
I20260927 04:03:24.792840 25587 consensus_queue.cc:260] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.793042 25587 ts_tablet_manager.cc:1428] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:24.793363 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.793241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:41933)
0927 04:03:24.793299 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.793333 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 04:03:24.793375 25590 tablet_bootstrap.cc:492] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: Bootstrap starting.
I20260927 04:03:24.793838 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.793685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:52497)
0927 04:03:24.793754 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.793781 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 04:03:24.793866 25590 tablet_bootstrap.cc:654] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:24.794008 25586 tablet_bootstrap.cc:492] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: No bootstrap required, opened a new log
I20260927 04:03:24.794080 25586 ts_tablet_manager.cc:1397] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 04:03:24.794198 25586 raft_consensus.cc:348] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.794325 25586 raft_consensus.cc:374] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:24.794351 25586 raft_consensus.cc:729] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84555cae7414ffc947a877ac4315649, State: Initialized, Role: FOLLOWER
I20260927 04:03:24.794417 25586 consensus_queue.cc:260] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.794639 25586 ts_tablet_manager.cc:1428] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:24.794884 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.794763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:41885)
0927 04:03:24.794813 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.794851 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 04:03:24.795018 25347 heartbeater.cc:503] Master 127.17.184.252:36099 was elected leader, sending a full tablet report...
I20260927 04:03:24.795197 25590 tablet_bootstrap.cc:492] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: No bootstrap required, opened a new log
I20260927 04:03:24.795251 25590 ts_tablet_manager.cc:1397] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:24.795349 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.795257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:36193)
0927 04:03:24.795296 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.795324 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 04:03:24.795413 25590 raft_consensus.cc:348] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.795492 25590 raft_consensus.cc:374] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:24.795518 25590 raft_consensus.cc:729] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d155427f4504ceca4f5a00791005a3b, State: Initialized, Role: FOLLOWER
I20260927 04:03:24.795570 25590 consensus_queue.cc:260] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.795740 25590 ts_tablet_manager.cc:1428] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:24.796013 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.795916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:50343)
0927 04:03:24.795961 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.795988 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 04:03:24.796268 25483 heartbeater.cc:503] Master 127.17.184.252:36099 was elected leader, sending a full tablet report...
I20260927 04:03:24.796332 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.796258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:45067)
0927 04:03:24.796291 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.796313 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260927 04:03:24.900257 25594 raft_consensus.cc:482] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:24.900362 25594 raft_consensus.cc:504] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.900648 25594 leader_election.cc:288] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969), 60d807011b38409097e30e028fb20cbb (127.17.184.193:33249)
I20260927 04:03:24.904896 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.900989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:37871 (local address 127.17.184.194:45969)
0927 04:03:24.901040 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:24.901042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:24.901056 (+ 14us) server_negotiation.cc:407] Connection header received
0927 04:03:24.901168 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:24.901170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:24.901180 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:24.901230 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:24.901454 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.902241 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.903914 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.904478 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.904559 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.904569 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:24.904770 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 04:03:24.904822 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 04:03:24.904907 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.900910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:45969 (local address 127.17.184.195:37871)
0927 04:03:24.900989 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.901016 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:24.901075 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:24.901289 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:24.901290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:24.901292 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:24.901374 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.901376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.902376 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.902376 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.903790 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.903794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.904575 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.904576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.904673 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.904678 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:03:24.904742 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 04:03:24.904777 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":77}
I20260927 04:03:24.904970 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.901174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:34647 (local address 127.17.184.193:33249)
0927 04:03:24.901219 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:24.901221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:24.901309 (+ 88us) server_negotiation.cc:407] Connection header received
0927 04:03:24.901402 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:24.901403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:24.901431 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:24.901487 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:24.901600 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.902288 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.903915 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.904478 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.904559 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.904569 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:24.904837 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 04:03:24.904894 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:03:24.905191 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.901027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33249 (local address 127.17.184.195:34647)
0927 04:03:24.901248 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.901257 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:24.901333 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:24.901508 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:24.901509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:24.901510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:24.901539 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.901540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.902373 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.902375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.903790 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:24.903794 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.904665 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.904666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:24.904746 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.904752 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 04:03:24.904812 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 04:03:24.904854 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":108,"thread_start_us":115,"threads_started":1}
I20260927 04:03:24.905213 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8352d7c385b848b0bc6818838afc3530" candidate_uuid: "5d155427f4504ceca4f5a00791005a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84555cae7414ffc947a877ac4315649" is_pre_election: true
I20260927 04:03:24.905334 25301 raft_consensus.cc:2454] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d155427f4504ceca4f5a00791005a3b in term 0.
I20260927 04:03:24.905440 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8352d7c385b848b0bc6818838afc3530" candidate_uuid: "5d155427f4504ceca4f5a00791005a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d807011b38409097e30e028fb20cbb" is_pre_election: true
I20260927 04:03:24.905494 25370 leader_election.cc:302] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d155427f4504ceca4f5a00791005a3b, e84555cae7414ffc947a877ac4315649; no voters:
I20260927 04:03:24.905593 25164 raft_consensus.cc:2454] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d155427f4504ceca4f5a00791005a3b in term 0.
I20260927 04:03:24.905622 25594 raft_consensus.cc:2781] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:24.905658 25594 raft_consensus.cc:482] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:24.905699 25594 raft_consensus.cc:3037] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:24.906440 25594 raft_consensus.cc:504] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.906582 25594 leader_election.cc:288] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [CANDIDATE]: Term 1 election: Requested vote from peers e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969), 60d807011b38409097e30e028fb20cbb (127.17.184.193:33249)
I20260927 04:03:24.906780 25164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8352d7c385b848b0bc6818838afc3530" candidate_uuid: "5d155427f4504ceca4f5a00791005a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d807011b38409097e30e028fb20cbb"
I20260927 04:03:24.906795 25301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8352d7c385b848b0bc6818838afc3530" candidate_uuid: "5d155427f4504ceca4f5a00791005a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84555cae7414ffc947a877ac4315649"
I20260927 04:03:24.906864 25164 raft_consensus.cc:3037] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:24.906864 25301 raft_consensus.cc:3037] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:24.907656 25301 raft_consensus.cc:2454] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d155427f4504ceca4f5a00791005a3b in term 1.
I20260927 04:03:24.907656 25164 raft_consensus.cc:2454] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d155427f4504ceca4f5a00791005a3b in term 1.
I20260927 04:03:24.907848 25370 leader_election.cc:302] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d155427f4504ceca4f5a00791005a3b, e84555cae7414ffc947a877ac4315649; no voters:
I20260927 04:03:24.908002 25594 raft_consensus.cc:2781] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:24.908185 25594 raft_consensus.cc:686] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 1 LEADER]: Becoming Leader. State: Replica: 5d155427f4504ceca4f5a00791005a3b, State: Running, Role: LEADER
I20260927 04:03:24.908249 25594 consensus_queue.cc:237] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } }
I20260927 04:03:24.908740 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.908557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:37305)
0927 04:03:24.908624 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.908664 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260927 04:03:24.909111 25023 catalog_manager.cc:5495] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d155427f4504ceca4f5a00791005a3b (127.17.184.195). New cstate: current_term: 1 leader_uuid: "5d155427f4504ceca4f5a00791005a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d155427f4504ceca4f5a00791005a3b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34693 } health_report { overall_health: HEALTHY } } }
I20260927 04:03:24.935086 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.932550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56110 (local address 127.17.184.195:34693)
0927 04:03:24.932600 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:24.932602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:24.932608 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:03:24.932644 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:24.932645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:24.932661 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:24.932769 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:24.932864 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.933426 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.934173 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:24.934457 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:24.934513 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:24.934656 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:24.934786 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:24.934914 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:24.934971 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 04:03:24.934993 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:03:24.936364 25301 raft_consensus.cc:1261] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 1 FOLLOWER]: Refusing update from remote peer 5d155427f4504ceca4f5a00791005a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:24.936437 25164 raft_consensus.cc:1261] T 8352d7c385b848b0bc6818838afc3530 P 60d807011b38409097e30e028fb20cbb [term 1 FOLLOWER]: Refusing update from remote peer 5d155427f4504ceca4f5a00791005a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:24.936681 25596 consensus_queue.cc:1035] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84555cae7414ffc947a877ac4315649" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:24.936832 25594 consensus_queue.cc:1035] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d807011b38409097e30e028fb20cbb" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:24.937884 25210 heartbeater.cc:503] Master 127.17.184.252:36099 was elected leader, sending a full tablet report...
I20260927 04:03:24.938012 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.937757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:58895)
0927 04:03:24.937927 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.937976 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260927 04:03:24.938090 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.937995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:52101)
0927 04:03:24.938028 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.938058 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 04:03:24.939037 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.938824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:55783)
0927 04:03:24.938974 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.939005 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260927 04:03:24.939083 25600 mvcc.cc:205] Tried to move back new op lower bound from 7333813473016643584 to 7333813472905175040. Current Snapshot: MvccSnapshot[applied={T|T < 7333813473016643584}]
I20260927 04:03:24.939411 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.938991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.194:49263)
0927 04:03:24.939358 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.939386 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353}
I20260927 04:03:24.940063 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.939976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.195:37279)
0927 04:03:24.940007 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.940032 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260927 04:03:24.940527 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.940392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:34757 (local address 127.17.184.193:57859)
0927 04:03:24.940474 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.940501 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:34757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 04:03:24.941267 25031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:24.941355 25031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:24.941521 25601 mvcc.cc:205] Tried to move back new op lower bound from 7333813473016643584 to 7333813472905175040. Current Snapshot: MvccSnapshot[applied={T|T < 7333813473016643584}]
I20260927 04:03:24.942571 25031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae095317e0aa47fb85ea3292e86dfe17: Sending DeleteTablet for 3 replicas of tablet 8352d7c385b848b0bc6818838afc3530
I20260927 04:03:24.942921 25417 tablet_service.cc:1514] Processing DeleteTablet for tablet 8352d7c385b848b0bc6818838afc3530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:24 UTC) from {username='oryx', principal='oryx/127.17.184.252@KRBTEST.COM'} at 127.0.0.1:56102
I20260927 04:03:24.942924 25281 tablet_service.cc:1514] Processing DeleteTablet for tablet 8352d7c385b848b0bc6818838afc3530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:24 UTC) from {username='oryx', principal='oryx/127.17.184.252@KRBTEST.COM'} at 127.0.0.1:40286
I20260927 04:03:24.943097 25144 tablet_service.cc:1514] Processing DeleteTablet for tablet 8352d7c385b848b0bc6818838afc3530 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:24 UTC) from {username='oryx', principal='oryx/127.17.184.252@KRBTEST.COM'} at 127.0.0.1:44234
I20260927 04:03:24.943430 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 24991
I20260927 04:03:24.945045 25610 tablet_replica.cc:330] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: stopping tablet replica
I20260927 04:03:24.945113 25610 raft_consensus.cc:2229] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:24.945188 25610 raft_consensus.cc:2258] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:24.945286 25609 tablet_replica.cc:330] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: stopping tablet replica
I20260927 04:03:24.945339 25609 raft_consensus.cc:2229] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:24.945384 25609 raft_consensus.cc:2258] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:24.945717 25610 ts_tablet_manager.cc:1905] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:24.945717 25609 ts_tablet_manager.cc:1905] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:24.947414 25609 ts_tablet_manager.cc:1918] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:24.947410 25610 ts_tablet_manager.cc:1918] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:24.947466 25610 log.cc:1196] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/8352d7c385b848b0bc6818838afc3530
I20260927 04:03:24.947466 25609 log.cc:1196] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/8352d7c385b848b0bc6818838afc3530
I20260927 04:03:24.947700 25609 ts_tablet_manager.cc:1939] T 8352d7c385b848b0bc6818838afc3530 P e84555cae7414ffc947a877ac4315649: Deleting consensus metadata
I20260927 04:03:24.947723 25610 ts_tablet_manager.cc:1939] T 8352d7c385b848b0bc6818838afc3530 P 5d155427f4504ceca4f5a00791005a3b: Deleting consensus metadata
I20260927 04:03:24.948247 25015 catalog_manager.cc:4841] TS e84555cae7414ffc947a877ac4315649 (127.17.184.194:45969): tablet 8352d7c385b848b0bc6818838afc3530 (table test-table [id=be11b014e4034940835faa2db0c81085]) successfully deleted
I20260927 04:03:24.948247 25016 catalog_manager.cc:4841] TS 5d155427f4504ceca4f5a00791005a3b (127.17.184.195:34693): tablet 8352d7c385b848b0bc6818838afc3530 (table test-table [id=be11b014e4034940835faa2db0c81085]) successfully deleted
I20260927 04:03:24.951140 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 25218
I20260927 04:03:24.951297 25583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:24.951142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33249 (local address 127.0.0.1:44248)
0927 04:03:24.951223 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:24.951261 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33249: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 04:03:24.951435 25015 catalog_manager.cc:4584] TS 60d807011b38409097e30e028fb20cbb (127.17.184.193:33249): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8352d7c385b848b0bc6818838afc3530: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33249: connect: Connection refused (error 111)
I20260927 04:03:24.957422 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 25354
I20260927 04:03:24.963028 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 24709
I20260927 04:03:24.968324 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 24897
2026-09-27T04:03:24Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6621 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:25 dist-test-slave-zqrn krb5kdc[25614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:25 dist-test-slave-zqrn krb5kdc[25614](info): set up 2 sockets
Sep 27 04:03:25 dist-test-slave-zqrn krb5kdc[25614](info): commencing operation
krb5kdc: starting...
W20260927 04:03:26.039670 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.254.keytab.
Entry for principal oryx/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.254.keytab.
I20260927 04:03:26.166172 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33655
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.254.keytab
--principal=oryx/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:33655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:26.248714 25627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:26.248955 25627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:26.248981 25627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:26.250119 25627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:26.250165 25627 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 04:03:26.250180 25627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:26.250195 25627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:26.250211 25627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:26.250224 25627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:26.251745 25627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:33655
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.254.keytab
--principal=oryx/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:26.251964 25627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:26.252152 25627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:26.257253 25632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:26.257222 25633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:26.257259 25627 server_base.cc:1034] running on GCE node
W20260927 04:03:26.257246 25635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:26.257799 25627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:26.258060 25627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:26.259198 25627 hybrid_clock.cc:648] HybridClock initialized: now 1790481806259180 us; error 38 us; skew 500 ppm
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:26.261919 25627 init.cc:377] Logged in from keytab as oryx/127.17.184.254@KRBTEST.COM (short username oryx)
I20260927 04:03:26.262346 25627 webserver.cc:460] Webserver started at http://127.17.184.254:41545/ using document root <none> and password file <none>
I20260927 04:03:26.262583 25627 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:26.262647 25627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:26.262795 25627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:26.263926 25627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "d80af4dfbc85409b9aa6d5940f464344"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "30b6a61414668d8cd2c5ffc374ad74f0"
server_key_iv: "0dcb3937771b18474588b04d5b973d33"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.264281 25627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "d80af4dfbc85409b9aa6d5940f464344"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "30b6a61414668d8cd2c5ffc374ad74f0"
server_key_iv: "0dcb3937771b18474588b04d5b973d33"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.265643 25627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:26.266511 25642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:26.266723 25627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:26.266855 25627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "d80af4dfbc85409b9aa6d5940f464344"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "30b6a61414668d8cd2c5ffc374ad74f0"
server_key_iv: "0dcb3937771b18474588b04d5b973d33"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.266961 25627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:26.281425 25627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:26.283821 25627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:26.283968 25627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:26.288586 25627 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:33655
I20260927 04:03:26.288708 25704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:33655 every 8 connection(s)
I20260927 04:03:26.289000 25627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:26.289727 25705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:26.291111 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 25627
I20260927 04:03:26.291229 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:26.291428 18147 external_mini_cluster.cc:1428] Setting key 1a9c8c3e3e4ca7a6f8efd5e95e875eda
I20260927 04:03:26.292567 25705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: Bootstrap starting.
I20260927 04:03:26.293322 25705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:26.293687 25705 log.cc:824] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:26.294385 25705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: No bootstrap required, opened a new log
I20260927 04:03:26.296693 25705 raft_consensus.cc:348] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80af4dfbc85409b9aa6d5940f464344" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33655 } }
I20260927 04:03:26.296859 25705 raft_consensus.cc:374] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:26.296883 25705 raft_consensus.cc:729] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80af4dfbc85409b9aa6d5940f464344, State: Initialized, Role: FOLLOWER
I20260927 04:03:26.297019 25705 consensus_queue.cc:260] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80af4dfbc85409b9aa6d5940f464344" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33655 } }
I20260927 04:03:26.297092 25705 raft_consensus.cc:388] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:26.297140 25705 raft_consensus.cc:482] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:26.297197 25705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:26.297735 25705 raft_consensus.cc:504] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80af4dfbc85409b9aa6d5940f464344" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33655 } }
I20260927 04:03:26.297811 25705 leader_election.cc:302] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80af4dfbc85409b9aa6d5940f464344; no voters:
I20260927 04:03:26.297960 25705 leader_election.cc:288] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:26.297989 25710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:26.298105 25710 raft_consensus.cc:686] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [term 1 LEADER]: Becoming Leader. State: Replica: d80af4dfbc85409b9aa6d5940f464344, State: Running, Role: LEADER
I20260927 04:03:26.298184 25710 consensus_queue.cc:237] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80af4dfbc85409b9aa6d5940f464344" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33655 } }
I20260927 04:03:26.298229 25705 sys_catalog.cc:564] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:26.298485 25710 sys_catalog.cc:455] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d80af4dfbc85409b9aa6d5940f464344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80af4dfbc85409b9aa6d5940f464344" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33655 } } }
I20260927 04:03:26.298552 25710 sys_catalog.cc:458] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [sys.catalog]: This master's current role is: LEADER
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:26.298873 25710 sys_catalog.cc:455] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d80af4dfbc85409b9aa6d5940f464344. Latest consensus state: current_term: 1 leader_uuid: "d80af4dfbc85409b9aa6d5940f464344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80af4dfbc85409b9aa6d5940f464344" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33655 } } }
I20260927 04:03:26.298965 25710 sys_catalog.cc:458] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344 [sys.catalog]: This master's current role is: LEADER
W20260927 04:03:26.300100 25726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:03:26.300155 25726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:03:26.300212 25714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:26.300427 25714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:26.302029 25714 catalog_manager.cc:1347] Generated new cluster ID: e78f4314c1c3423aa9d2469f5de15069
I20260927 04:03:26.302079 25714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:26.303542 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.292014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.17.184.254:33655)
0927 04:03:26.292191 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.292200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.292233 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:03:26.292626 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.292632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.292717 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.292855 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.293696 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.294769 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.295914 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.296364 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.296459 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.299543 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:26.299548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:26.299548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:26.299561 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:26.301725 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.302195 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.302198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.302199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.302313 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.302541 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.302542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.302542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.302805 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:26.303008 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.303145 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.303317 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260927 04:03:26.315527 25714 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:26.316115 25714 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:26.318346 25714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: Generated new TSK 0
I20260927 04:03:26.318543 25714 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab.
Entry for principal oryx/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab.
I20260927 04:03:26.350687 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33655
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab
--principal=oryx/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:26.437690 25731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:26.437904 25731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:26.437930 25731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:26.439064 25731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:26.439098 25731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:26.439118 25731 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:03:26.439136 25731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:26.440551 25731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab
--principal=oryx/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:33655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:26.440730 25731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:26.440922 25731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:26.445998 25737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:26.446100 25739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:26.446058 25731 server_base.cc:1034] running on GCE node
W20260927 04:03:26.445998 25736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:26.446419 25731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:26.446671 25731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:26.447873 25731 hybrid_clock.cc:648] HybridClock initialized: now 1790481806447854 us; error 45 us; skew 500 ppm
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:26.450588 25731 init.cc:377] Logged in from keytab as oryx/127.17.184.193@KRBTEST.COM (short username oryx)
I20260927 04:03:26.451151 25731 webserver.cc:460] Webserver started at http://127.17.184.193:40983/ using document root <none> and password file <none>
I20260927 04:03:26.451401 25731 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:26.451468 25731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:26.451612 25731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:26.452709 25731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "1c38ee2d5f3ef4b059a7b94cf2331bda"
server_key_iv: "6f43d77547d541e270bd2b1815432da5"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.453068 25731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "1c38ee2d5f3ef4b059a7b94cf2331bda"
server_key_iv: "6f43d77547d541e270bd2b1815432da5"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.454499 25731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:26.455483 25746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:26.455856 25731 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:26.455971 25731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "1c38ee2d5f3ef4b059a7b94cf2331bda"
server_key_iv: "6f43d77547d541e270bd2b1815432da5"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.456063 25731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:26.475658 25731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:26.477737 25731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:26.477886 25731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:26.478519 25731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:26.478559 25731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:26.478585 25731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:26.478632 25731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:26.485528 25731 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:45591
I20260927 04:03:26.485610 25861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:45591 every 8 connection(s)
I20260927 04:03:26.485924 25731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:26.486532 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 25731
I20260927 04:03:26.486649 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:26.486832 18147 external_mini_cluster.cc:1428] Setting key 3612c4077514de9a738d9366d81931f0
I20260927 04:03:26.490125 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.479111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33655 (local address 127.17.184.193:50017)
0927 04:03:26.479286 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.479322 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.479868 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.480126 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.480130 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.480510 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:26.480824 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.480832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.481947 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.481948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.483348 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.483351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.483880 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.483881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.483980 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.484733 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:26.484744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:26.486500 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:26.488429 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.488434 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.488436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.488650 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.488911 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.488912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.488913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.489067 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.489372 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:26.489380 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:26.489520 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.489651 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.489849 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260927 04:03:26.490500 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.479109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:50017 (local address 127.17.184.254:33655)
0927 04:03:26.479173 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.479175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.479403 (+ 228us) server_negotiation.cc:407] Connection header received
0927 04:03:26.479973 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.479975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.479988 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.480046 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.480945 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.481830 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.483438 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.483780 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.483880 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.486603 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:26.486605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:26.486605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:26.486614 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:26.488343 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.488729 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.488730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.488731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.488819 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.489169 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.489170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.489170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.489291 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:26.489370 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.490264 (+ 894us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.490362 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 04:03:26.490923 25862 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33655
I20260927 04:03:26.491011 25862 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:26.491202 25862 heartbeater.cc:511] Master 127.17.184.254:33655 requested a full tablet report, sending...
I20260927 04:03:26.491775 25658 ts_manager.cc:194] Registered new tserver with Master: f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591)
I20260927 04:03:26.492491 25658 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:50017
WARNING: no policy specified for oryx/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.194@KRBTEST.COM" created.
I20260927 04:03:26.501744 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.492052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:42345 (local address 127.17.184.254:33655)
0927 04:03:26.492086 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.492088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.495105 (+ 3017us) server_negotiation.cc:407] Connection header received
0927 04:03:26.495192 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.495193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.495203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.495253 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.495566 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.496049 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.496743 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.496952 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.497033 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.497967 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:26.497971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:26.497971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:26.497978 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:26.500578 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.500943 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.500945 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.500945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.501002 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.501204 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.501206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.501206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.501329 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:26.501398 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.501560 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.501644 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:03:26.501755 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.491995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33655 (local address 127.17.184.193:42345)
0927 04:03:26.495034 (+ 3039us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.495044 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.495118 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.495255 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.495256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.495437 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:26.495491 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.495492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.496145 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.496146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.496658 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.496659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.497034 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.497035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.497106 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.497563 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:26.497568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:26.497861 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:26.500676 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.500678 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.500678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.500857 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.501078 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.501079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.501079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.501139 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.501400 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:26.501401 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:26.501452 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.501532 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.501645 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3016,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab.
Entry for principal oryx/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab.
I20260927 04:03:26.514765 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33655
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab
--principal=oryx/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:26.599256 25871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:26.599552 25871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:26.599581 25871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:26.600831 25871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:26.600875 25871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:26.600896 25871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:03:26.600914 25871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:26.602407 25871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab
--principal=oryx/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:33655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:26.602636 25871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:26.602844 25871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:26.608006 25876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:26.607966 25877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:26.607963 25879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:26.608337 25871 server_base.cc:1034] running on GCE node
I20260927 04:03:26.608606 25871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:26.608886 25871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:26.610049 25871 hybrid_clock.cc:648] HybridClock initialized: now 1790481806610042 us; error 36 us; skew 500 ppm
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:26.612730 25871 init.cc:377] Logged in from keytab as oryx/127.17.184.194@KRBTEST.COM (short username oryx)
I20260927 04:03:26.613204 25871 webserver.cc:460] Webserver started at http://127.17.184.194:35905/ using document root <none> and password file <none>
I20260927 04:03:26.613440 25871 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:26.613503 25871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:26.613651 25871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:26.614701 25871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "e78038c5f7da45b799eaf1502bda9ce0"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "096d03f95fb9263e8eecb42971f1d4c7"
server_key_iv: "02dedf06536f7f81a667df9e53112995"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.615089 25871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "e78038c5f7da45b799eaf1502bda9ce0"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "096d03f95fb9263e8eecb42971f1d4c7"
server_key_iv: "02dedf06536f7f81a667df9e53112995"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.616417 25871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:26.617144 25886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:26.617318 25871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:26.617413 25871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "e78038c5f7da45b799eaf1502bda9ce0"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "096d03f95fb9263e8eecb42971f1d4c7"
server_key_iv: "02dedf06536f7f81a667df9e53112995"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.617487 25871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:26.629773 25871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:26.632416 25871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:26.632592 25871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:26.633131 25871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:26.633190 25871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:26.633261 25871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:26.633297 25871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:26.640736 25871 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:41359
I20260927 04:03:26.640834 26001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:41359 every 8 connection(s)
I20260927 04:03:26.641175 25871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:26.643088 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.633784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:40175 (local address 127.17.184.254:33655)
0927 04:03:26.633848 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.633851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.634069 (+ 218us) server_negotiation.cc:407] Connection header received
0927 04:03:26.634681 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.634683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.634698 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.634755 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.635473 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.635987 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.637188 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.637407 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.637494 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.640189 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:26.640191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:26.640191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:26.640200 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:26.641841 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.642247 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.642248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.642249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.642320 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.642548 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.642549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.642549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.642658 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:26.642711 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.642947 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.643025 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:03:26.643338 25895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.633708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33655 (local address 127.17.184.194:40175)
0927 04:03:26.633949 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.633985 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.634554 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.634844 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.634849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.635165 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:26.635380 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.635385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.636088 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.636089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.637075 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.637079 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.637505 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.637506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.637626 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.638198 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:26.638212 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:26.640086 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:26.641911 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.641915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.641916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.642131 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.642388 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.642389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.642389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.642482 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.642702 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:26.642704 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:26.642810 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.642885 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.643064 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260927 04:03:26.643946 26002 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33655
I20260927 04:03:26.644032 26002 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:26.644230 26002 heartbeater.cc:511] Master 127.17.184.254:33655 requested a full tablet report, sending...
I20260927 04:03:26.644593 25658 ts_manager.cc:194] Registered new tserver with Master: e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359)
I20260927 04:03:26.645098 25658 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:40175
I20260927 04:03:26.650225 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 25871
I20260927 04:03:26.650353 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:26.650588 18147 external_mini_cluster.cc:1428] Setting key 234729d375930c14a4c69e035bdbfeed
I20260927 04:03:26.651358 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.644964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:37335 (local address 127.17.184.254:33655)
0927 04:03:26.644999 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.645001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.645343 (+ 342us) server_negotiation.cc:407] Connection header received
0927 04:03:26.645400 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.645401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.645412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.645480 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.645942 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.646417 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.647297 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.647479 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.647551 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.648541 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:26.648542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:26.648542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:26.648548 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:26.649942 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.650361 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.650363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.650363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.650432 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.650635 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.650636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.650636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.650879 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:26.651010 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.651192 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.651264 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 04:03:26.651376 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.644903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33655 (local address 127.17.184.194:37335)
0927 04:03:26.645275 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.645288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.645351 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.645525 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.645526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.645736 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:26.645827 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.645829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.646535 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.646536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.647206 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.647208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.647576 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.647577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.647694 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.648139 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:26.648144 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:26.648460 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:26.650044 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.650045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.650045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.650266 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.650494 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.650495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.650495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.650565 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.651017 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:26.651018 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:26.651068 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.651153 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.651266 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab.
Entry for principal oryx/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab.
I20260927 04:03:26.678740 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33655
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab
--principal=oryx/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:26.761610 26011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:26.761838 26011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:26.761863 26011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:26.763023 26011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:26.763060 26011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:26.763080 26011 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 04:03:26.763098 26011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:26.764648 26011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab
--principal=oryx/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.184.254:33655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:26.764911 26011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:26.765146 26011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:26.770385 26019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:26.770443 26017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:26.770385 26016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:26.770625 26011 server_base.cc:1034] running on GCE node
I20260927 04:03:26.770990 26011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:26.771252 26011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:26.772414 26011 hybrid_clock.cc:648] HybridClock initialized: now 1790481806772404 us; error 32 us; skew 500 ppm
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:26.775238 26011 init.cc:377] Logged in from keytab as oryx/127.17.184.195@KRBTEST.COM (short username oryx)
I20260927 04:03:26.775815 26011 webserver.cc:460] Webserver started at http://127.17.184.195:46487/ using document root <none> and password file <none>
I20260927 04:03:26.776063 26011 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:26.776110 26011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:26.776254 26011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:26.777371 26011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "019aecf88ef147a38cb64fd9cdd7f6a3"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "22c3557295f036c86c0013a0a6de7c68"
server_key_iv: "c55aa780359e6718a9817bfb5140693a"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.777741 26011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "019aecf88ef147a38cb64fd9cdd7f6a3"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "22c3557295f036c86c0013a0a6de7c68"
server_key_iv: "c55aa780359e6718a9817bfb5140693a"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.779264 26011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:26.780087 26026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:26.780287 26011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:26.780375 26011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "019aecf88ef147a38cb64fd9cdd7f6a3"
format_stamp: "Formatted at 2026-09-27 04:03:26 on dist-test-slave-zqrn"
server_key: "22c3557295f036c86c0013a0a6de7c68"
server_key_iv: "c55aa780359e6718a9817bfb5140693a"
server_key_version: "encryptionkey@0"
I20260927 04:03:26.780445 26011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:26.789366 26011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:26.791698 26011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:26.791844 26011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:26.792474 26011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:26.792526 26011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:26.792555 26011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:26.792589 26011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:26.800048 26011 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:40689
I20260927 04:03:26.800134 26141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:40689 every 8 connection(s)
I20260927 04:03:26.800475 26011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:26.803156 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.792889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:50767 (local address 127.17.184.254:33655)
0927 04:03:26.792938 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.792940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.793139 (+ 199us) server_negotiation.cc:407] Connection header received
0927 04:03:26.793782 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.793784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.793799 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.793867 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.794614 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.795348 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.796466 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.796682 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.796772 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.800348 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:26.800350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:26.800350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:26.800358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:26.801825 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.802181 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.802182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.802182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.802233 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.802438 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.802438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.802438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.802542 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:26.802592 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.803002 (+ 410us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.803075 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:03:26.803310 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.792873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33655 (local address 127.17.184.195:50767)
0927 04:03:26.793020 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.793058 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.793673 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.793958 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.793962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.794285 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:26.794522 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.794527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.795451 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.795452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.796369 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.796371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.796782 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.796784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.796915 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.797475 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:26.797487 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:26.800253 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:26.801898 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.801903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.801905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.802114 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.802289 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.802290 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.802290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.802386 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.802659 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:26.802665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:26.802802 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.802965 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.803113 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260927 04:03:26.803874 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 26011
I20260927 04:03:26.803982 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:26.804020 26142 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33655
I20260927 04:03:26.804105 26142 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:26.804172 18147 external_mini_cluster.cc:1428] Setting key 08e97f58bfda1ce2462a398a8cf45642
I20260927 04:03:26.804308 26142 heartbeater.cc:511] Master 127.17.184.254:33655 requested a full tablet report, sending...
I20260927 04:03:26.804667 25658 ts_manager.cc:194] Registered new tserver with Master: 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:26.805042 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:03:26.805092 25658 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:50767
I20260927 04:03:26.811124 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.805262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:44113 (local address 127.17.184.254:33655)
0927 04:03:26.805331 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.805333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.805469 (+ 136us) server_negotiation.cc:407] Connection header received
0927 04:03:26.805538 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.805539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.805549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.805597 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.805914 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.806385 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.807415 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.807596 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.807656 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.808576 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:26.808578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:26.808578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:26.808585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:26.809946 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.810298 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.810299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.810300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.810352 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.810543 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.810544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.810544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.810681 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:26.810754 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.810999 (+ 245us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.811067 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 04:03:26.811199 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.805180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33655 (local address 127.17.184.195:44113)
0927 04:03:26.805402 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.805411 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.805481 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.805608 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.805609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.805784 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:26.805833 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.805835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.806478 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.806479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.807337 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.807339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.807669 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.807670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.807741 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.808194 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:26.808199 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:26.808473 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:26.810043 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.810044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.810044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.810218 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.810420 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:26.810421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:26.810421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:26.810478 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:26.810783 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:26.810784 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:26.810832 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.810952 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.811075 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 04:03:26.829493 25708 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:26.821516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.17.184.254:33655)
0927 04:03:26.821654 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.821655 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.821673 (+ 18us) server_negotiation.cc:407] Connection header received
0927 04:03:26.821695 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.821696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.821705 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.821764 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.822304 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.822826 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.823640 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.823869 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.824294 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.829429 (+ 5135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33650: BlockingRecv error: recv got EOF from 127.0.0.1:33650 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Sep 27 04:03:26 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:26.839944 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.830345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33658 (local address 127.17.184.254:33655)
0927 04:03:26.830410 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.830412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.830477 (+ 65us) server_negotiation.cc:407] Connection header received
0927 04:03:26.830519 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.830520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.830529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.830575 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:26.831095 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.831919 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.833145 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.833479 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.833557 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.836738 (+ 3181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:26.836740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:26.836740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:26.836748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:26.838476 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.838981 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.838982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.838983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.839056 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:26.839308 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:26.839309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:26.839310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:26.839457 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:26.839558 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.839732 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.839813 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:03:26.847146 25658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:26.849040 25658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:26.855845 26167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.851630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:45591 (local address 127.0.0.1:41360)
0927 04:03:26.852025 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.852032 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.852077 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.852326 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.852328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.852330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:26.852392 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.852394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.853753 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.853754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.854830 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.854832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.855533 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.855534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.855624 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.855638 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.855707 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.855760 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":10,"thread_start_us":142,"threads_started":1}
I20260927 04:03:26.855983 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.851696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37818 (local address 127.17.184.195:40689)
0927 04:03:26.852185 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.852188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.852195 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:03:26.852247 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.852250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.852265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.852517 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:26.852576 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.853251 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.854747 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.855602 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.855725 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.855786 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.855817 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.855861 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":87,"threads_started":1}
I20260927 04:03:26.856226 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.851784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.17.184.193:45591)
0927 04:03:26.851955 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.851960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.852088 (+ 128us) server_negotiation.cc:407] Connection header received
0927 04:03:26.852132 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.852134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.852152 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.852242 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:26.852480 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.853673 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.854946 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.855437 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.855551 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.855613 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.856026 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.856147 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260927 04:03:26.855924 26164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.851466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:40689 (local address 127.0.0.1:37818)
0927 04:03:26.851912 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.851921 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.851957 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.852373 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.852374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.852376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:26.852442 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.852444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.853343 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.853344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.854662 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.854665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.855688 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.855688 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.855751 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.855754 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.855794 (+ 40us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.855824 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":97,"threads_started":1}
I20260927 04:03:26.856086 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.851420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:41359 (local address 127.0.0.1:59352)
0927 04:03:26.851698 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.851746 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.851814 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.852086 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.852089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.852092 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:26.852175 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.852178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.853715 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.853716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.854863 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.854866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.855853 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.855854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.855916 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.855921 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.855956 (+ 35us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.856001 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":35,"thread_start_us":65,"threads_started":1}
I20260927 04:03:26.856627 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.851534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.17.184.194:41359)
0927 04:03:26.851901 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.851906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.851917 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:26.851958 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.851960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.851977 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.852063 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:26.852304 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.853603 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.854985 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.855751 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.856357 (+ 606us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.856425 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.856454 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.856499 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":67,"threads_started":1}
I20260927 04:03:26.857036 25796 tablet_service.cc:1467] Processing CreateTablet for tablet 1e41c6ecc8054262a49fd1619a8be2e1 (DEFAULT_TABLE table=test-table [id=4db542c2d27647a6af39e0dd6435f7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:26.857095 26076 tablet_service.cc:1467] Processing CreateTablet for tablet 1e41c6ecc8054262a49fd1619a8be2e1 (DEFAULT_TABLE table=test-table [id=4db542c2d27647a6af39e0dd6435f7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:26.857367 25796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e41c6ecc8054262a49fd1619a8be2e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:26.857486 26076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e41c6ecc8054262a49fd1619a8be2e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:26.859560 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 1e41c6ecc8054262a49fd1619a8be2e1 (DEFAULT_TABLE table=test-table [id=4db542c2d27647a6af39e0dd6435f7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:26.859952 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e41c6ecc8054262a49fd1619a8be2e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:26.860723 26170 tablet_bootstrap.cc:492] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Bootstrap starting.
I20260927 04:03:26.861146 26171 tablet_bootstrap.cc:492] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: Bootstrap starting.
I20260927 04:03:26.861271 26170 tablet_bootstrap.cc:654] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:26.861495 26170 log.cc:824] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:26.861801 26171 tablet_bootstrap.cc:654] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:26.862131 26170 tablet_bootstrap.cc:492] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: No bootstrap required, opened a new log
I20260927 04:03:26.862133 26171 log.cc:824] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:26.862178 26170 ts_tablet_manager.cc:1397] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:26.863008 26171 tablet_bootstrap.cc:492] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: No bootstrap required, opened a new log
I20260927 04:03:26.863080 26171 ts_tablet_manager.cc:1397] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:26.863902 26170 raft_consensus.cc:348] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:26.864054 26170 raft_consensus.cc:374] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:26.864097 26170 raft_consensus.cc:729] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b1fd54392a4c8a9a187e1751d5d8d8, State: Initialized, Role: FOLLOWER
I20260927 04:03:26.864229 26170 consensus_queue.cc:260] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:26.864507 26170 ts_tablet_manager.cc:1428] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:26.864601 26175 tablet_bootstrap.cc:492] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Bootstrap starting.
I20260927 04:03:26.864656 25862 heartbeater.cc:503] Master 127.17.184.254:33655 was elected leader, sending a full tablet report...
I20260927 04:03:26.865131 26175 tablet_bootstrap.cc:654] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:26.865428 26175 log.cc:824] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:26.865777 26171 raft_consensus.cc:348] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:26.865907 26171 raft_consensus.cc:374] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:26.865931 26171 raft_consensus.cc:729] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019aecf88ef147a38cb64fd9cdd7f6a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:26.866078 26171 consensus_queue.cc:260] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:26.866184 26175 tablet_bootstrap.cc:492] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: No bootstrap required, opened a new log
I20260927 04:03:26.866247 26175 ts_tablet_manager.cc:1397] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:26.866390 26171 ts_tablet_manager.cc:1428] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:03:26.866427 26142 heartbeater.cc:503] Master 127.17.184.254:33655 was elected leader, sending a full tablet report...
I20260927 04:03:26.868072 26175 raft_consensus.cc:348] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:26.868224 26175 raft_consensus.cc:374] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:26.868269 26175 raft_consensus.cc:729] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78038c5f7da45b799eaf1502bda9ce0, State: Initialized, Role: FOLLOWER
I20260927 04:03:26.868383 26175 consensus_queue.cc:260] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:26.868620 26175 ts_tablet_manager.cc:1428] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 04:03:26.868705 26002 heartbeater.cc:503] Master 127.17.184.254:33655 was elected leader, sending a full tablet report...
W20260927 04:03:26.891764 26003 tablet.cc:2369] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:03:26.986768 25863 tablet.cc:2369] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:26.994982 26174 raft_consensus.cc:482] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:26.995118 26174 raft_consensus.cc:504] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:26.995551 26174 leader_election.cc:288] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359), 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:26.998726 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.995957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:38543 (local address 127.17.184.195:40689)
0927 04:03:26.996004 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.996006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.996019 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:03:26.996263 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.996264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.996274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.996322 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:26.996483 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.996947 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.997928 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.998308 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.998441 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.998452 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.998606 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.998643 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:03:26.998734 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.995876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:50055 (local address 127.17.184.194:41359)
0927 04:03:26.995941 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:26.995944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:26.996006 (+ 62us) server_negotiation.cc:407] Connection header received
0927 04:03:26.996126 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:26.996127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:26.996135 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:26.996180 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:26.996392 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.996968 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.997957 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.998379 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.998441 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.998452 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:26.998612 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 04:03:26.998658 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:03:26.998751 26179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.995813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:40689 (local address 127.17.184.193:38543)
0927 04:03:26.995965 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.995976 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.996034 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.996344 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.996345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.996347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:26.996413 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.996415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.997023 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.997023 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.997838 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.997840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.998441 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.998442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.998529 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.998535 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.998600 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.998643 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260927 04:03:26.998770 26180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:26.995775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:41359 (local address 127.17.184.193:50055)
0927 04:03:26.995939 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:26.995950 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:26.996037 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:26.996200 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:26.996201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:26.996203 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:26.996275 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.996276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.997070 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.997071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.997845 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:26.997847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:26.998460 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:26.998461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:26.998529 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:26.998532 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:26.998602 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:03:26.998647 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260927 04:03:26.999097 26096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e41c6ecc8054262a49fd1619a8be2e1" candidate_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" is_pre_election: true
I20260927 04:03:26.999223 26096 raft_consensus.cc:2454] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b1fd54392a4c8a9a187e1751d5d8d8 in term 0.
I20260927 04:03:26.999473 25750 leader_election.cc:302] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019aecf88ef147a38cb64fd9cdd7f6a3, f9b1fd54392a4c8a9a187e1751d5d8d8; no voters:
I20260927 04:03:26.999598 26174 raft_consensus.cc:2781] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:26.999576 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e41c6ecc8054262a49fd1619a8be2e1" candidate_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e78038c5f7da45b799eaf1502bda9ce0" is_pre_election: true
I20260927 04:03:26.999668 26174 raft_consensus.cc:482] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:26.999698 26174 raft_consensus.cc:3037] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:26.999722 25956 raft_consensus.cc:2454] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b1fd54392a4c8a9a187e1751d5d8d8 in term 0.
I20260927 04:03:27.000519 26174 raft_consensus.cc:504] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.000666 26174 leader_election.cc:288] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [CANDIDATE]: Term 1 election: Requested vote from peers e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359), 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:27.000877 26096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e41c6ecc8054262a49fd1619a8be2e1" candidate_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3"
I20260927 04:03:27.000881 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e41c6ecc8054262a49fd1619a8be2e1" candidate_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e78038c5f7da45b799eaf1502bda9ce0"
I20260927 04:03:27.000963 25956 raft_consensus.cc:3037] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.000963 26096 raft_consensus.cc:3037] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.001874 26096 raft_consensus.cc:2454] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b1fd54392a4c8a9a187e1751d5d8d8 in term 1.
I20260927 04:03:27.001874 25956 raft_consensus.cc:2454] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b1fd54392a4c8a9a187e1751d5d8d8 in term 1.
I20260927 04:03:27.002030 25750 leader_election.cc:302] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019aecf88ef147a38cb64fd9cdd7f6a3, f9b1fd54392a4c8a9a187e1751d5d8d8; no voters:
I20260927 04:03:27.002142 26174 raft_consensus.cc:2781] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:27.002259 26174 raft_consensus.cc:686] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 LEADER]: Becoming Leader. State: Replica: f9b1fd54392a4c8a9a187e1751d5d8d8, State: Running, Role: LEADER
I20260927 04:03:27.002355 26174 consensus_queue.cc:237] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.003136 25658 catalog_manager.cc:5495] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:27.037400 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.034857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41364 (local address 127.17.184.193:45591)
0927 04:03:27.034935 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.034938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.034948 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:27.034993 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.034994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.035013 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.035109 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:27.035233 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.035705 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.036514 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.036705 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.036763 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.036980 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.037105 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.037210 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.037227 (+ 17us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.037251 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:03:27.039546 25956 raft_consensus.cc:1261] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Refusing update from remote peer f9b1fd54392a4c8a9a187e1751d5d8d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.039585 26096 raft_consensus.cc:1261] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Refusing update from remote peer f9b1fd54392a4c8a9a187e1751d5d8d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.040014 26181 consensus_queue.cc:1035] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.040195 26174 consensus_queue.cc:1035] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.042637 26186 mvcc.cc:205] Tried to move back new op lower bound from 7333813481631068160 to 7333813481482223616. Current Snapshot: MvccSnapshot[applied={T|T < 7333813481631068160}]
I20260927 04:03:27.043503 26185 mvcc.cc:205] Tried to move back new op lower bound from 7333813481631068160 to 7333813481482223616. Current Snapshot: MvccSnapshot[applied={T|T < 7333813481631068160}]
I20260927 04:03:27.045894 25658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:27.046015 25658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:27.046770 25658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: Sending DeleteTablet for 3 replicas of tablet 1e41c6ecc8054262a49fd1619a8be2e1
I20260927 04:03:27.047179 25796 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e41c6ecc8054262a49fd1619a8be2e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41360
I20260927 04:03:27.047175 26076 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e41c6ecc8054262a49fd1619a8be2e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:37818
I20260927 04:03:27.047576 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e41c6ecc8054262a49fd1619a8be2e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:59352
I20260927 04:03:27.047808 26196 tablet_replica.cc:330] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: stopping tablet replica
I20260927 04:03:27.047926 26196 raft_consensus.cc:2229] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:27.047943 26195 tablet_replica.cc:330] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: stopping tablet replica
I20260927 04:03:27.048020 26195 raft_consensus.cc:2229] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:27.048051 26196 raft_consensus.cc:2258] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.048137 26195 raft_consensus.cc:2258] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.048192 25658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:27.048341 25658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:27.048532 26196 ts_tablet_manager.cc:1905] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.048532 26195 ts_tablet_manager.cc:1905] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.050076 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 8567941a2ee54dfca81d8d51bcc793b9 (DEFAULT_TABLE table=test-table [id=452568e368e64fc19dce7414bb52af0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.050226 26196 ts_tablet_manager.cc:1918] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:27.050199 26076 tablet_service.cc:1467] Processing CreateTablet for tablet 8567941a2ee54dfca81d8d51bcc793b9 (DEFAULT_TABLE table=test-table [id=452568e368e64fc19dce7414bb52af0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.050246 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8567941a2ee54dfca81d8d51bcc793b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.050282 26196 log.cc:1196] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/1e41c6ecc8054262a49fd1619a8be2e1
I20260927 04:03:27.050387 26076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8567941a2ee54dfca81d8d51bcc793b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.050536 26196 ts_tablet_manager.cc:1939] T 1e41c6ecc8054262a49fd1619a8be2e1 P e78038c5f7da45b799eaf1502bda9ce0: Deleting consensus metadata
I20260927 04:03:27.051005 25644 catalog_manager.cc:4841] TS e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359): tablet 1e41c6ecc8054262a49fd1619a8be2e1 (table test-table [id=4db542c2d27647a6af39e0dd6435f7b9]) successfully deleted
I20260927 04:03:27.051440 26175 tablet_bootstrap.cc:492] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: Bootstrap starting.
I20260927 04:03:27.051487 25796 tablet_service.cc:1467] Processing CreateTablet for tablet 8567941a2ee54dfca81d8d51bcc793b9 (DEFAULT_TABLE table=test-table [id=452568e368e64fc19dce7414bb52af0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.051581 25796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8567941a2ee54dfca81d8d51bcc793b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.051831 26175 tablet_bootstrap.cc:654] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.052278 26171 tablet_bootstrap.cc:492] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: Bootstrap starting.
I20260927 04:03:27.052663 26170 tablet_bootstrap.cc:492] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: Bootstrap starting.
I20260927 04:03:27.052698 26171 tablet_bootstrap.cc:654] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.052963 26195 ts_tablet_manager.cc:1918] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:27.053023 26195 log.cc:1196] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/1e41c6ecc8054262a49fd1619a8be2e1
I20260927 04:03:27.053076 26170 tablet_bootstrap.cc:654] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.053275 26195 ts_tablet_manager.cc:1939] T 1e41c6ecc8054262a49fd1619a8be2e1 P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting consensus metadata
I20260927 04:03:27.053485 26194 tablet_replica.cc:330] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: stopping tablet replica
I20260927 04:03:27.053597 26194 raft_consensus.cc:2229] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:27.053726 26171 tablet_bootstrap.cc:492] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: No bootstrap required, opened a new log
I20260927 04:03:27.053776 26171 ts_tablet_manager.cc:1397] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 04:03:27.053922 26171 raft_consensus.cc:348] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.054013 26171 raft_consensus.cc:374] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.054039 26171 raft_consensus.cc:729] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019aecf88ef147a38cb64fd9cdd7f6a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.054045 25644 catalog_manager.cc:4841] TS f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591): tablet 1e41c6ecc8054262a49fd1619a8be2e1 (table test-table [id=4db542c2d27647a6af39e0dd6435f7b9]) successfully deleted
I20260927 04:03:27.054107 26171 consensus_queue.cc:260] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.054203 26171 ts_tablet_manager.cc:1428] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:27.053730 26194 raft_consensus.cc:2258] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.054375 26170 tablet_bootstrap.cc:492] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: No bootstrap required, opened a new log
I20260927 04:03:27.054411 26170 ts_tablet_manager.cc:1397] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:27.054549 26170 raft_consensus.cc:348] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.054636 26170 raft_consensus.cc:374] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.054658 26170 raft_consensus.cc:729] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b1fd54392a4c8a9a187e1751d5d8d8, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.054698 26170 consensus_queue.cc:260] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.054788 26170 ts_tablet_manager.cc:1428] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:27.054873 26194 ts_tablet_manager.cc:1905] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.056332 26175 tablet_bootstrap.cc:492] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: No bootstrap required, opened a new log
I20260927 04:03:27.056391 26175 ts_tablet_manager.cc:1397] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 04:03:27.056416 26194 ts_tablet_manager.cc:1918] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:27.056486 26194 log.cc:1196] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/1e41c6ecc8054262a49fd1619a8be2e1
I20260927 04:03:27.056553 26175 raft_consensus.cc:348] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.056655 26175 raft_consensus.cc:374] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.056684 26175 raft_consensus.cc:729] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78038c5f7da45b799eaf1502bda9ce0, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.056730 26175 consensus_queue.cc:260] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.056774 26194 ts_tablet_manager.cc:1939] T 1e41c6ecc8054262a49fd1619a8be2e1 P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting consensus metadata
I20260927 04:03:27.056838 26175 ts_tablet_manager.cc:1428] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:27.057181 25646 catalog_manager.cc:4841] TS 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689): tablet 1e41c6ecc8054262a49fd1619a8be2e1 (table test-table [id=4db542c2d27647a6af39e0dd6435f7b9]) successfully deleted
I20260927 04:03:27.227777 26191 raft_consensus.cc:482] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:27.227900 26191 raft_consensus.cc:504] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.228111 26191 leader_election.cc:288] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359), 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:27.228385 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8567941a2ee54dfca81d8d51bcc793b9" candidate_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e78038c5f7da45b799eaf1502bda9ce0" is_pre_election: true
I20260927 04:03:27.228400 26096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8567941a2ee54dfca81d8d51bcc793b9" candidate_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" is_pre_election: true
I20260927 04:03:27.228498 26096 raft_consensus.cc:2454] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b1fd54392a4c8a9a187e1751d5d8d8 in term 0.
I20260927 04:03:27.228504 25956 raft_consensus.cc:2454] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b1fd54392a4c8a9a187e1751d5d8d8 in term 0.
I20260927 04:03:27.228662 25750 leader_election.cc:302] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019aecf88ef147a38cb64fd9cdd7f6a3, f9b1fd54392a4c8a9a187e1751d5d8d8; no voters:
I20260927 04:03:27.228770 26191 raft_consensus.cc:2781] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:27.228812 26191 raft_consensus.cc:482] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:27.228830 26191 raft_consensus.cc:3037] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.229642 26191 raft_consensus.cc:504] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.229786 26191 leader_election.cc:288] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [CANDIDATE]: Term 1 election: Requested vote from peers e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359), 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:27.229938 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8567941a2ee54dfca81d8d51bcc793b9" candidate_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e78038c5f7da45b799eaf1502bda9ce0"
I20260927 04:03:27.230028 25956 raft_consensus.cc:3037] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.230029 26096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8567941a2ee54dfca81d8d51bcc793b9" candidate_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3"
I20260927 04:03:27.230116 26096 raft_consensus.cc:3037] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.230683 25956 raft_consensus.cc:2454] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b1fd54392a4c8a9a187e1751d5d8d8 in term 1.
I20260927 04:03:27.230789 26096 raft_consensus.cc:2454] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b1fd54392a4c8a9a187e1751d5d8d8 in term 1.
I20260927 04:03:27.230865 25748 leader_election.cc:302] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e78038c5f7da45b799eaf1502bda9ce0, f9b1fd54392a4c8a9a187e1751d5d8d8; no voters:
I20260927 04:03:27.230973 26191 raft_consensus.cc:2781] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:27.231019 26191 raft_consensus.cc:686] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 LEADER]: Becoming Leader. State: Replica: f9b1fd54392a4c8a9a187e1751d5d8d8, State: Running, Role: LEADER
I20260927 04:03:27.231096 26191 consensus_queue.cc:237] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.231676 25659 catalog_manager.cc:5495] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } health_report { overall_health: UNKNOWN } } }
Sep 27 04:03:27 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481806, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.254@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:27.263245 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.255759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.17.184.254:33655)
0927 04:03:27.255881 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.255884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.255993 (+ 109us) server_negotiation.cc:407] Connection header received
0927 04:03:27.256094 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.256094 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.256109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.256166 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:27.256619 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.257153 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.257859 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.258042 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.258134 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.260370 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:27.260371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:27.260372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:27.260378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:27.261933 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:27.262361 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:27.262362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:27.262363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:27.262421 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:27.262618 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:27.262619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:27.262620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:27.262737 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:27.262817 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.263003 (+ 186us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.263122 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 04:03:27.263345 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.255688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33655 (local address 127.0.0.1:33662)
0927 04:03:27.255913 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.255929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.256014 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.256170 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.256171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.256485 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:27.256533 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.256535 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.257273 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.257274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.257784 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.257785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.258133 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.258134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.258183 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.258751 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:27.258757 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:27.260275 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:27.262049 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:27.262057 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:27.262060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:27.262270 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:27.262488 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:27.262489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:27.262489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:27.262552 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:27.262807 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:27.262815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:27.262864 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.262977 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.263190 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":100,"thread_start_us":118,"threads_started":1}
I20260927 04:03:27.265492 25659 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:33662:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 04:03:27.265683 25659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:27.267295 25796 tablet_service.cc:1467] Processing CreateTablet for tablet 74a414b5546d4f66836c53a6ef21147e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5991b824f3a449cae1c76215ecfe8af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:03:27.267349 26076 tablet_service.cc:1467] Processing CreateTablet for tablet 74a414b5546d4f66836c53a6ef21147e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5991b824f3a449cae1c76215ecfe8af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:03:27.267396 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 74a414b5546d4f66836c53a6ef21147e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d5991b824f3a449cae1c76215ecfe8af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 04:03:27.267469 25796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a414b5546d4f66836c53a6ef21147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.267514 26076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a414b5546d4f66836c53a6ef21147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.267522 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74a414b5546d4f66836c53a6ef21147e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.269007 26171 tablet_bootstrap.cc:492] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: Bootstrap starting.
I20260927 04:03:27.269008 26175 tablet_bootstrap.cc:492] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: Bootstrap starting.
I20260927 04:03:27.269480 26171 tablet_bootstrap.cc:654] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.269480 26175 tablet_bootstrap.cc:654] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.269850 26170 tablet_bootstrap.cc:492] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: Bootstrap starting.
I20260927 04:03:27.270123 26175 tablet_bootstrap.cc:492] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: No bootstrap required, opened a new log
I20260927 04:03:27.270208 26175 ts_tablet_manager.cc:1397] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:27.270285 26171 tablet_bootstrap.cc:492] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: No bootstrap required, opened a new log
I20260927 04:03:27.270311 26170 tablet_bootstrap.cc:654] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.270340 26171 ts_tablet_manager.cc:1397] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:27.270381 26175 raft_consensus.cc:348] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.270464 26175 raft_consensus.cc:374] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.270489 26175 raft_consensus.cc:729] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78038c5f7da45b799eaf1502bda9ce0, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.270502 26171 raft_consensus.cc:348] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.270568 26171 raft_consensus.cc:374] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.270588 26171 raft_consensus.cc:729] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019aecf88ef147a38cb64fd9cdd7f6a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.270552 26175 consensus_queue.cc:260] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.270651 26171 consensus_queue.cc:260] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.270668 26175 ts_tablet_manager.cc:1428] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:27.270757 26171 ts_tablet_manager.cc:1428] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:27.270814 26177 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.270808 26178 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.270936 26178 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:27.270936 26177 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:27.271157 26170 tablet_bootstrap.cc:492] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: No bootstrap required, opened a new log
I20260927 04:03:27.271194 26170 ts_tablet_manager.cc:1397] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:27.271292 26170 raft_consensus.cc:348] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.271363 26170 raft_consensus.cc:374] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.271381 26170 raft_consensus.cc:729] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b1fd54392a4c8a9a187e1751d5d8d8, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.271426 26170 consensus_queue.cc:260] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.271517 26170 ts_tablet_manager.cc:1428] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:27.271545 26191 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.271653 26191 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:27.290086 26178 raft_consensus.cc:482] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:27.290211 26178 raft_consensus.cc:504] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.290642 26178 leader_election.cc:288] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591), 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:27.293780 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.291086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:46305 (local address 127.17.184.193:45591)
0927 04:03:27.291132 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.291135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.291198 (+ 63us) server_negotiation.cc:407] Connection header received
0927 04:03:27.291303 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.291305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.291318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.291389 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:27.291541 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.292072 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.293053 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.293430 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.293508 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.293518 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.293672 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.293713 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:03:27.293799 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.290805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:45591 (local address 127.17.184.194:46305)
0927 04:03:27.291139 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.291149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.291221 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.291392 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.291393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.291395 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:27.291463 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.291465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.292147 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.292148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.292952 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.292954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.293507 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.293508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.293585 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.293589 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.293667 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.293708 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":74,"threads_started":1}
I20260927 04:03:27.293882 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.290984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:40799 (local address 127.17.184.195:40689)
0927 04:03:27.291033 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.291035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.291180 (+ 145us) server_negotiation.cc:407] Connection header received
0927 04:03:27.291253 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.291255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.291266 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.291331 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:27.291564 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.292110 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.293113 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.293507 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.293611 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.293619 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.293766 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.293811 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:03:27.293896 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.290795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:40689 (local address 127.17.184.194:40799)
0927 04:03:27.291082 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.291095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.291186 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.291406 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.291407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.291410 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:27.291482 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.291483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.292198 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.292206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.293014 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.293016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.293614 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.293615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.293689 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.293692 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.293761 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.293806 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":62,"threads_started":1}
I20260927 04:03:27.294131 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a414b5546d4f66836c53a6ef21147e" candidate_uuid: "e78038c5f7da45b799eaf1502bda9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" is_pre_election: true
I20260927 04:03:27.294284 25816 raft_consensus.cc:2454] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e78038c5f7da45b799eaf1502bda9ce0 in term 0.
I20260927 04:03:27.294454 25888 leader_election.cc:302] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e78038c5f7da45b799eaf1502bda9ce0, f9b1fd54392a4c8a9a187e1751d5d8d8; no voters:
I20260927 04:03:27.294461 26096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a414b5546d4f66836c53a6ef21147e" candidate_uuid: "e78038c5f7da45b799eaf1502bda9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" is_pre_election: true
I20260927 04:03:27.294543 26096 raft_consensus.cc:2454] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e78038c5f7da45b799eaf1502bda9ce0 in term 0.
I20260927 04:03:27.294587 26178 raft_consensus.cc:2781] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:27.294626 26178 raft_consensus.cc:482] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:27.294662 26178 raft_consensus.cc:3037] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.295426 26178 raft_consensus.cc:504] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.295576 26178 leader_election.cc:288] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [CANDIDATE]: Term 1 election: Requested vote from peers f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591), 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:27.295738 26096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a414b5546d4f66836c53a6ef21147e" candidate_uuid: "e78038c5f7da45b799eaf1502bda9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3"
I20260927 04:03:27.295814 26096 raft_consensus.cc:3037] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.295778 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74a414b5546d4f66836c53a6ef21147e" candidate_uuid: "e78038c5f7da45b799eaf1502bda9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8"
I20260927 04:03:27.295876 25816 raft_consensus.cc:3037] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.296389 26096 raft_consensus.cc:2454] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e78038c5f7da45b799eaf1502bda9ce0 in term 1.
I20260927 04:03:27.296581 25890 leader_election.cc:302] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019aecf88ef147a38cb64fd9cdd7f6a3, e78038c5f7da45b799eaf1502bda9ce0; no voters:
I20260927 04:03:27.296649 25816 raft_consensus.cc:2454] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e78038c5f7da45b799eaf1502bda9ce0 in term 1.
I20260927 04:03:27.296677 26178 raft_consensus.cc:2781] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:27.296780 26178 raft_consensus.cc:686] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [term 1 LEADER]: Becoming Leader. State: Replica: e78038c5f7da45b799eaf1502bda9ce0, State: Running, Role: LEADER
I20260927 04:03:27.296846 26178 consensus_queue.cc:237] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.297425 26212 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: TxnStatusTablet state changed. Reason: New leader e78038c5f7da45b799eaf1502bda9ce0. Latest consensus state: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.297516 26212 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:03:27.297585 25658 catalog_manager.cc:5495] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:27.297724 26214 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 04:03:27.297823 26214 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260927 04:03:27.302412 26143 tablet.cc:2369] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:27.310452 26206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.307679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:41359 (local address 127.0.0.1:59354)
0927 04:03:27.307759 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.307772 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.307828 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.308039 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.308040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.308041 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:27.308093 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.308096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.308766 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.308767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.309416 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.309418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.309842 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.309843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.309935 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.309951 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.310247 (+ 296us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.310257 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.310316 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.310374 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:03:27.310441 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.307768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.17.184.194:41359)
0927 04:03:27.307848 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.307850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.307878 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:03:27.307914 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.307915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.307933 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.308030 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:27.308165 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.308681 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.309508 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.309741 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.309840 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.310051 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.310177 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.310264 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.310330 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.310352 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:03:27.311810 26096 raft_consensus.cc:1261] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Refusing update from remote peer e78038c5f7da45b799eaf1502bda9ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.311997 25816 raft_consensus.cc:1261] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Refusing update from remote peer e78038c5f7da45b799eaf1502bda9ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.312101 26178 consensus_queue.cc:1035] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.312302 26178 consensus_queue.cc:1035] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.312666 26177 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: TxnStatusTablet state changed. Reason: New leader e78038c5f7da45b799eaf1502bda9ce0. Latest consensus state: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.312764 26177 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:27.313063 26191 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: TxnStatusTablet state changed. Reason: New leader e78038c5f7da45b799eaf1502bda9ce0. Latest consensus state: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.313163 26191 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:27.313541 26178 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.313628 26178 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:03:27.314227 26177 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.314329 26177 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P 019aecf88ef147a38cb64fd9cdd7f6a3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:27.315439 26212 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.315526 26212 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P e78038c5f7da45b799eaf1502bda9ce0: This TxnStatusTablet replica's current role is: LEADER
I20260927 04:03:27.315663 26184 mvcc.cc:205] Tried to move back new op lower bound from 7333813482747183104 to 7333813482688528384. Current Snapshot: MvccSnapshot[applied={T|T < 7333813482747183104}]
I20260927 04:03:27.315896 26191 tablet_replica.cc:439] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } } }
I20260927 04:03:27.315977 26191 tablet_replica.cc:442] T 74a414b5546d4f66836c53a6ef21147e P f9b1fd54392a4c8a9a187e1751d5d8d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 04:03:27.324430 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.321312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:54277 (local address 127.17.184.194:41359)
0927 04:03:27.321362 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.321364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.321536 (+ 172us) server_negotiation.cc:407] Connection header received
0927 04:03:27.321623 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.321624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.321634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.321689 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:27.321845 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.322413 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.323723 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.323916 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.323981 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.324192 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.324222 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.324259 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.324354 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.324372 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:03:27.324506 26221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.321237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:41359 (local address 127.17.184.193:54277)
0927 04:03:27.321446 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.321459 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.321533 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.321709 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.321710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.321712 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:27.321774 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.321776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.322493 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.322494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.323605 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.323609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.323998 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.324000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.324066 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.324087 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.324286 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.324288 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.324349 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.324406 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":97,"threads_started":1}
I20260927 04:03:27.327170 26096 raft_consensus.cc:1261] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Refusing update from remote peer f9b1fd54392a4c8a9a187e1751d5d8d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.327247 25956 raft_consensus.cc:1261] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Refusing update from remote peer f9b1fd54392a4c8a9a187e1751d5d8d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.327409 26191 consensus_queue.cc:1035] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.327497 26174 consensus_queue.cc:1035] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.336889 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.333705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:56973 (local address 127.17.184.193:45591)
0927 04:03:27.333750 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.333752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.333828 (+ 76us) server_negotiation.cc:407] Connection header received
0927 04:03:27.333945 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.333946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.333956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.334016 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:27.334247 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.335087 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.335977 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.336331 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.336408 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.336600 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.336640 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.336697 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.336776 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.336800 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:03:27.337260 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.333588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:45591 (local address 127.17.184.194:56973)
0927 04:03:27.333765 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.333776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.333856 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.334059 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.334060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.334061 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:27.334172 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.334174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.335169 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.335170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.335908 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.335911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.336417 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.336418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.336498 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.336514 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.336720 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.336722 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.336760 (+ 38us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.336808 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":107,"threads_started":1}
I20260927 04:03:27.346159 25659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:27.346284 25659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:27.346972 25659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: Sending DeleteTablet for 3 replicas of tablet 8567941a2ee54dfca81d8d51bcc793b9
I20260927 04:03:27.347254 26076 tablet_service.cc:1514] Processing DeleteTablet for tablet 8567941a2ee54dfca81d8d51bcc793b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:37818
I20260927 04:03:27.347254 25796 tablet_service.cc:1514] Processing DeleteTablet for tablet 8567941a2ee54dfca81d8d51bcc793b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41360
I20260927 04:03:27.347379 26195 tablet_replica.cc:330] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: stopping tablet replica
I20260927 04:03:27.347434 26194 tablet_replica.cc:330] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: stopping tablet replica
I20260927 04:03:27.347462 26195 raft_consensus.cc:2229] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:27.347507 26194 raft_consensus.cc:2229] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:27.347539 26195 raft_consensus.cc:2258] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.347553 26194 raft_consensus.cc:2258] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.347856 26195 ts_tablet_manager.cc:1905] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.347882 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet 8567941a2ee54dfca81d8d51bcc793b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:59352
I20260927 04:03:27.347992 26196 tablet_replica.cc:330] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: stopping tablet replica
I20260927 04:03:27.348045 26196 raft_consensus.cc:2229] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:27.348089 26196 raft_consensus.cc:2258] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.348373 26196 ts_tablet_manager.cc:1905] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.348451 26194 ts_tablet_manager.cc:1905] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.349666 26195 ts_tablet_manager.cc:1918] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:27.349721 26195 log.cc:1196] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/8567941a2ee54dfca81d8d51bcc793b9
I20260927 04:03:27.349957 26194 ts_tablet_manager.cc:1918] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:27.349996 26195 ts_tablet_manager.cc:1939] T 8567941a2ee54dfca81d8d51bcc793b9 P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting consensus metadata
I20260927 04:03:27.350011 26194 log.cc:1196] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/8567941a2ee54dfca81d8d51bcc793b9
I20260927 04:03:27.350255 26194 ts_tablet_manager.cc:1939] T 8567941a2ee54dfca81d8d51bcc793b9 P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting consensus metadata
I20260927 04:03:27.350332 26196 ts_tablet_manager.cc:1918] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:27.350353 25644 catalog_manager.cc:4841] TS f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591): tablet 8567941a2ee54dfca81d8d51bcc793b9 (table test-table [id=452568e368e64fc19dce7414bb52af0d]) successfully deleted
I20260927 04:03:27.350384 26196 log.cc:1196] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/8567941a2ee54dfca81d8d51bcc793b9
I20260927 04:03:27.350639 26196 ts_tablet_manager.cc:1939] T 8567941a2ee54dfca81d8d51bcc793b9 P e78038c5f7da45b799eaf1502bda9ce0: Deleting consensus metadata
I20260927 04:03:27.351001 25644 catalog_manager.cc:4841] TS e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359): tablet 8567941a2ee54dfca81d8d51bcc793b9 (table test-table [id=452568e368e64fc19dce7414bb52af0d]) successfully deleted
I20260927 04:03:27.353140 25646 catalog_manager.cc:4841] TS 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689): tablet 8567941a2ee54dfca81d8d51bcc793b9 (table test-table [id=452568e368e64fc19dce7414bb52af0d]) successfully deleted
Sep 27 04:03:27 dist-test-slave-zqrn krb5kdc[25614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 04:03:27 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:27 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 04:03:27.377503 25708 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:27.371013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33666 (local address 127.17.184.254:33655)
0927 04:03:27.371074 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.371077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.371186 (+ 109us) server_negotiation.cc:407] Connection header received
0927 04:03:27.371273 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.371274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.371286 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.371347 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:27.372067 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.372873 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.373676 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.373982 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.374058 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.377426 (+ 3368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33666: BlockingRecv error: recv got EOF from 127.0.0.1:33666 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 04:03:27 dist-test-slave-zqrn krb5kdc[25614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM
I20260927 04:03:27.386790 25708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.378467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.17.184.254:33655)
0927 04:03:27.378524 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.378526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.378650 (+ 124us) server_negotiation.cc:407] Connection header received
0927 04:03:27.378708 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.378709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.378718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.378811 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:27.379434 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.380000 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.381004 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.381202 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.381265 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.383968 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:27.383969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:27.383970 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:27.383979 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:27.385455 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:27.385906 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:27.385907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:27.385907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:27.385974 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:27.386182 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:27.386183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:27.386184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:27.386319 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:27.386428 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.386588 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.386673 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 04:03:27.388413 25659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:27.388594 25659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:27.390190 26076 tablet_service.cc:1467] Processing CreateTablet for tablet e24797c9580f4f759f9d47e0892c6a8a (DEFAULT_TABLE table=test-table [id=7a52831152624f04a8d634493c90d45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.390235 25936 tablet_service.cc:1467] Processing CreateTablet for tablet e24797c9580f4f759f9d47e0892c6a8a (DEFAULT_TABLE table=test-table [id=7a52831152624f04a8d634493c90d45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.390237 25796 tablet_service.cc:1467] Processing CreateTablet for tablet e24797c9580f4f759f9d47e0892c6a8a (DEFAULT_TABLE table=test-table [id=7a52831152624f04a8d634493c90d45c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.390347 26076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24797c9580f4f759f9d47e0892c6a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.390347 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24797c9580f4f759f9d47e0892c6a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.390380 25796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e24797c9580f4f759f9d47e0892c6a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.391808 26171 tablet_bootstrap.cc:492] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: Bootstrap starting.
I20260927 04:03:27.391944 26170 tablet_bootstrap.cc:492] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: Bootstrap starting.
I20260927 04:03:27.392036 26175 tablet_bootstrap.cc:492] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: Bootstrap starting.
I20260927 04:03:27.392235 26171 tablet_bootstrap.cc:654] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.392235 26170 tablet_bootstrap.cc:654] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.392417 26175 tablet_bootstrap.cc:654] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.393105 26175 tablet_bootstrap.cc:492] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: No bootstrap required, opened a new log
I20260927 04:03:27.393162 26175 ts_tablet_manager.cc:1397] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:27.393361 26170 tablet_bootstrap.cc:492] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: No bootstrap required, opened a new log
I20260927 04:03:27.393339 26175 raft_consensus.cc:348] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.393404 26170 ts_tablet_manager.cc:1397] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:27.393414 26175 raft_consensus.cc:374] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.393437 26175 raft_consensus.cc:729] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78038c5f7da45b799eaf1502bda9ce0, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.393492 26175 consensus_queue.cc:260] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.393517 26170 raft_consensus.cc:348] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.393582 26170 raft_consensus.cc:374] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.393606 26170 raft_consensus.cc:729] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b1fd54392a4c8a9a187e1751d5d8d8, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.393613 26175 ts_tablet_manager.cc:1428] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:27.393656 26170 consensus_queue.cc:260] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.393760 26170 ts_tablet_manager.cc:1428] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:27.395377 26171 tablet_bootstrap.cc:492] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: No bootstrap required, opened a new log
I20260927 04:03:27.395454 26171 ts_tablet_manager.cc:1397] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 04:03:27.395614 26171 raft_consensus.cc:348] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.395710 26171 raft_consensus.cc:374] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.395745 26171 raft_consensus.cc:729] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019aecf88ef147a38cb64fd9cdd7f6a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.395797 26171 consensus_queue.cc:260] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.395898 26171 ts_tablet_manager.cc:1428] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:27.482003 26178 raft_consensus.cc:482] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:27.482103 26178 raft_consensus.cc:504] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.482295 26178 leader_election.cc:288] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591), 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:27.482496 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24797c9580f4f759f9d47e0892c6a8a" candidate_uuid: "e78038c5f7da45b799eaf1502bda9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" is_pre_election: true
I20260927 04:03:27.482568 26096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24797c9580f4f759f9d47e0892c6a8a" candidate_uuid: "e78038c5f7da45b799eaf1502bda9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" is_pre_election: true
I20260927 04:03:27.482615 25816 raft_consensus.cc:2454] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e78038c5f7da45b799eaf1502bda9ce0 in term 0.
I20260927 04:03:27.482653 26096 raft_consensus.cc:2454] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e78038c5f7da45b799eaf1502bda9ce0 in term 0.
I20260927 04:03:27.482879 25890 leader_election.cc:302] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019aecf88ef147a38cb64fd9cdd7f6a3, e78038c5f7da45b799eaf1502bda9ce0; no voters:
I20260927 04:03:27.483024 26178 raft_consensus.cc:2781] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:27.483074 26178 raft_consensus.cc:482] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:27.483106 26178 raft_consensus.cc:3037] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.483809 26178 raft_consensus.cc:504] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.483954 26178 leader_election.cc:288] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [CANDIDATE]: Term 1 election: Requested vote from peers f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591), 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689)
I20260927 04:03:27.484092 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24797c9580f4f759f9d47e0892c6a8a" candidate_uuid: "e78038c5f7da45b799eaf1502bda9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8"
I20260927 04:03:27.484117 26096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e24797c9580f4f759f9d47e0892c6a8a" candidate_uuid: "e78038c5f7da45b799eaf1502bda9ce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3"
I20260927 04:03:27.484192 26096 raft_consensus.cc:3037] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.484196 25816 raft_consensus.cc:3037] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.484817 25816 raft_consensus.cc:2454] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e78038c5f7da45b799eaf1502bda9ce0 in term 1.
I20260927 04:03:27.484833 26096 raft_consensus.cc:2454] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e78038c5f7da45b799eaf1502bda9ce0 in term 1.
I20260927 04:03:27.485003 25888 leader_election.cc:302] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e78038c5f7da45b799eaf1502bda9ce0, f9b1fd54392a4c8a9a187e1751d5d8d8; no voters:
I20260927 04:03:27.485105 26178 raft_consensus.cc:2781] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:27.485157 26178 raft_consensus.cc:686] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 1 LEADER]: Becoming Leader. State: Replica: e78038c5f7da45b799eaf1502bda9ce0, State: Running, Role: LEADER
I20260927 04:03:27.485213 26178 consensus_queue.cc:237] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.486096 25659 catalog_manager.cc:5495] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "e78038c5f7da45b799eaf1502bda9ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:27.512727 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.510408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.17.184.194:41359)
0927 04:03:27.510478 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.510480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.510490 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:27.510539 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.510540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.510555 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.510603 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:27.510727 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.511291 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.512040 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.512241 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.512332 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.512467 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.512505 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.512566 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.512616 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.512640 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:03:27.513607 25816 raft_consensus.cc:1261] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Refusing update from remote peer e78038c5f7da45b799eaf1502bda9ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.513624 26096 raft_consensus.cc:1261] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Refusing update from remote peer e78038c5f7da45b799eaf1502bda9ce0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.513854 26178 consensus_queue.cc:1035] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.513953 26222 consensus_queue.cc:1035] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.516956 25659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:27.517068 25659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:27.517655 25659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: Sending DeleteTablet for 3 replicas of tablet e24797c9580f4f759f9d47e0892c6a8a
I20260927 04:03:27.517887 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet e24797c9580f4f759f9d47e0892c6a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:59352
I20260927 04:03:27.517921 26076 tablet_service.cc:1514] Processing DeleteTablet for tablet e24797c9580f4f759f9d47e0892c6a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:37818
I20260927 04:03:27.517930 25796 tablet_service.cc:1514] Processing DeleteTablet for tablet e24797c9580f4f759f9d47e0892c6a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41360
I20260927 04:03:27.517971 26196 tablet_replica.cc:330] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: stopping tablet replica
I20260927 04:03:27.517997 26194 tablet_replica.cc:330] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: stopping tablet replica
I20260927 04:03:27.518018 26195 tablet_replica.cc:330] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: stopping tablet replica
I20260927 04:03:27.518023 26196 raft_consensus.cc:2229] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:27.518034 26194 raft_consensus.cc:2229] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:27.518060 26195 raft_consensus.cc:2229] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:27.518071 26194 raft_consensus.cc:2258] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.518107 26196 raft_consensus.cc:2258] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.518111 26195 raft_consensus.cc:2258] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.518478 25659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 04:03:27.518541 26194 ts_tablet_manager.cc:1905] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.518548 26195 ts_tablet_manager.cc:1905] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 04:03:27.518610 25659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:27.519966 26076 tablet_service.cc:1467] Processing CreateTablet for tablet 94c33cc627a142b9903e04bb7f5b6bfe (DEFAULT_TABLE table=test-table [id=0ccdceb69293404d9e327daca106d280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.520265 25796 tablet_service.cc:1467] Processing CreateTablet for tablet 94c33cc627a142b9903e04bb7f5b6bfe (DEFAULT_TABLE table=test-table [id=0ccdceb69293404d9e327daca106d280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.520342 25936 tablet_service.cc:1467] Processing CreateTablet for tablet 94c33cc627a142b9903e04bb7f5b6bfe (DEFAULT_TABLE table=test-table [id=0ccdceb69293404d9e327daca106d280]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:27.520462 25936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c33cc627a142b9903e04bb7f5b6bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.520551 26196 ts_tablet_manager.cc:1905] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.520594 26194 ts_tablet_manager.cc:1918] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:27.520638 26194 log.cc:1196] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/e24797c9580f4f759f9d47e0892c6a8a
I20260927 04:03:27.520907 26194 ts_tablet_manager.cc:1939] T e24797c9580f4f759f9d47e0892c6a8a P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting consensus metadata
I20260927 04:03:27.521046 26076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c33cc627a142b9903e04bb7f5b6bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.521862 26175 tablet_bootstrap.cc:492] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: Bootstrap starting.
I20260927 04:03:27.522137 26196 ts_tablet_manager.cc:1918] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:27.522185 26175 tablet_bootstrap.cc:654] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.522286 26171 tablet_bootstrap.cc:492] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: Bootstrap starting.
I20260927 04:03:27.522192 26196 log.cc:1196] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/e24797c9580f4f759f9d47e0892c6a8a
I20260927 04:03:27.522701 26195 ts_tablet_manager.cc:1918] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:27.522711 26196 ts_tablet_manager.cc:1939] T e24797c9580f4f759f9d47e0892c6a8a P e78038c5f7da45b799eaf1502bda9ce0: Deleting consensus metadata
I20260927 04:03:27.522754 26195 log.cc:1196] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/e24797c9580f4f759f9d47e0892c6a8a
I20260927 04:03:27.523001 26195 ts_tablet_manager.cc:1939] T e24797c9580f4f759f9d47e0892c6a8a P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting consensus metadata
I20260927 04:03:27.523051 26171 tablet_bootstrap.cc:654] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.523062 25644 catalog_manager.cc:4841] TS e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359): tablet e24797c9580f4f759f9d47e0892c6a8a (table test-table [id=7a52831152624f04a8d634493c90d45c]) successfully deleted
I20260927 04:03:27.523132 25796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94c33cc627a142b9903e04bb7f5b6bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:27.523589 25646 catalog_manager.cc:4841] TS 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689): tablet e24797c9580f4f759f9d47e0892c6a8a (table test-table [id=7a52831152624f04a8d634493c90d45c]) successfully deleted
I20260927 04:03:27.523851 26175 tablet_bootstrap.cc:492] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: No bootstrap required, opened a new log
I20260927 04:03:27.523881 26175 ts_tablet_manager.cc:1397] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:27.523965 26175 raft_consensus.cc:348] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.524010 26175 raft_consensus.cc:374] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.524024 26175 raft_consensus.cc:729] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e78038c5f7da45b799eaf1502bda9ce0, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.524085 26175 consensus_queue.cc:260] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.524173 26175 ts_tablet_manager.cc:1428] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:27.524289 26170 tablet_bootstrap.cc:492] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8: Bootstrap starting.
I20260927 04:03:27.524544 26170 tablet_bootstrap.cc:654] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:27.524693 25644 catalog_manager.cc:4841] TS f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591): tablet e24797c9580f4f759f9d47e0892c6a8a (table test-table [id=7a52831152624f04a8d634493c90d45c]) successfully deleted
I20260927 04:03:27.525115 26170 tablet_bootstrap.cc:492] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8: No bootstrap required, opened a new log
I20260927 04:03:27.525162 26170 ts_tablet_manager.cc:1397] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:27.525255 26170 raft_consensus.cc:348] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.525324 26170 raft_consensus.cc:374] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.525352 26170 raft_consensus.cc:729] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b1fd54392a4c8a9a187e1751d5d8d8, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.525390 26170 consensus_queue.cc:260] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.525478 26170 ts_tablet_manager.cc:1428] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:27.527338 26171 tablet_bootstrap.cc:492] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: No bootstrap required, opened a new log
I20260927 04:03:27.527393 26171 ts_tablet_manager.cc:1397] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260927 04:03:27.527544 26171 raft_consensus.cc:348] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.527632 26171 raft_consensus.cc:374] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:27.527663 26171 raft_consensus.cc:729] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019aecf88ef147a38cb64fd9cdd7f6a3, State: Initialized, Role: FOLLOWER
I20260927 04:03:27.527709 26171 consensus_queue.cc:260] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.527806 26171 ts_tablet_manager.cc:1428] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260927 04:03:27.579052 26177 raft_consensus.cc:482] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:27.579180 26177 raft_consensus.cc:504] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.579545 26177 leader_election.cc:288] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591), e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359)
I20260927 04:03:27.583453 26251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.580038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:41359 (local address 127.17.184.195:39293)
0927 04:03:27.580274 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.580284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.580337 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.580506 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.580507 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.580508 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:27.580561 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.580562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.581412 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.581413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.582467 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.582469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.583121 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.583122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.583223 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.583228 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.583316 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.583358 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":53,"threads_started":1}
I20260927 04:03:27.583446 26169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.579922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:49055 (local address 127.17.184.193:45591)
0927 04:03:27.579987 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.579989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.580051 (+ 62us) server_negotiation.cc:407] Connection header received
0927 04:03:27.580182 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.580184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.580197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.580229 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:27.580473 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.581018 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.582484 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.583056 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.583145 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.583155 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.583340 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.583387 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 04:03:27.583454 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.580189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:39293 (local address 127.17.184.194:41359)
0927 04:03:27.580234 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.580236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.580349 (+ 113us) server_negotiation.cc:407] Connection header received
0927 04:03:27.580398 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.580399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.580409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.580463 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:27.580635 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.581296 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.582546 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.583026 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.583104 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.583116 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.583343 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.583385 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:03:27.583686 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c33cc627a142b9903e04bb7f5b6bfe" candidate_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e78038c5f7da45b799eaf1502bda9ce0" is_pre_election: true
I20260927 04:03:27.583808 25956 raft_consensus.cc:2454] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019aecf88ef147a38cb64fd9cdd7f6a3 in term 0.
I20260927 04:03:27.583935 26250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.579859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:45591 (local address 127.17.184.195:49055)
0927 04:03:27.579986 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.579997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.580083 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.580307 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.580308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.580310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:27.580398 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.580400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.581097 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.581098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.582386 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.582389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.583171 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.583172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.583249 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.583252 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.583751 (+ 499us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.583822 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":16,"thread_start_us":66,"threads_started":1}
I20260927 04:03:27.583986 26028 leader_election.cc:302] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019aecf88ef147a38cb64fd9cdd7f6a3, e78038c5f7da45b799eaf1502bda9ce0; no voters:
I20260927 04:03:27.584143 26177 raft_consensus.cc:2781] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:27.584185 26177 raft_consensus.cc:482] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:27.584206 26177 raft_consensus.cc:3037] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.584280 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c33cc627a142b9903e04bb7f5b6bfe" candidate_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" is_pre_election: true
I20260927 04:03:27.584363 25816 raft_consensus.cc:2454] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019aecf88ef147a38cb64fd9cdd7f6a3 in term 0.
I20260927 04:03:27.584997 26177 raft_consensus.cc:504] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.585117 26177 leader_election.cc:288] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [CANDIDATE]: Term 1 election: Requested vote from peers f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591), e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359)
I20260927 04:03:27.585282 25956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c33cc627a142b9903e04bb7f5b6bfe" candidate_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e78038c5f7da45b799eaf1502bda9ce0"
I20260927 04:03:27.585278 25816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94c33cc627a142b9903e04bb7f5b6bfe" candidate_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8"
I20260927 04:03:27.585357 25956 raft_consensus.cc:3037] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.585361 25816 raft_consensus.cc:3037] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:27.585925 25956 raft_consensus.cc:2454] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019aecf88ef147a38cb64fd9cdd7f6a3 in term 1.
I20260927 04:03:27.586076 26028 leader_election.cc:302] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019aecf88ef147a38cb64fd9cdd7f6a3, e78038c5f7da45b799eaf1502bda9ce0; no voters:
I20260927 04:03:27.586162 26177 raft_consensus.cc:2781] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:27.586186 25816 raft_consensus.cc:2454] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019aecf88ef147a38cb64fd9cdd7f6a3 in term 1.
I20260927 04:03:27.586265 26177 raft_consensus.cc:686] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 LEADER]: Becoming Leader. State: Replica: 019aecf88ef147a38cb64fd9cdd7f6a3, State: Running, Role: LEADER
I20260927 04:03:27.586331 26177 consensus_queue.cc:237] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } }
I20260927 04:03:27.587100 25659 catalog_manager.cc:5495] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195). New cstate: current_term: 1 leader_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "019aecf88ef147a38cb64fd9cdd7f6a3" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40689 } health_report { overall_health: HEALTHY } } }
I20260927 04:03:27.601845 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.599021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.17.184.195:40689)
0927 04:03:27.599064 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.599066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.599099 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:03:27.599168 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.599169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.599192 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.599277 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:27.599404 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.600139 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.600984 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.601285 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.601363 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.601491 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.601623 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.601700 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.601742 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.601764 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:03:27.606688 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.604282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:55191 (local address 127.17.184.194:41359)
0927 04:03:27.604330 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.604333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.604464 (+ 131us) server_negotiation.cc:407] Connection header received
0927 04:03:27.604530 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.604531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.604548 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.604601 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:27.604715 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.605226 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.605999 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.606202 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.606271 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.606441 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.606476 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.606534 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.606589 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.606608 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:03:27.606712 26256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.604226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:41359 (local address 127.17.184.195:55191)
0927 04:03:27.604399 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.604408 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.604479 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.604601 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.604602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.604603 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:27.604660 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.604661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.605295 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.605297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.605914 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.605916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.606281 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.606282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.606348 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.606366 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.606535 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.606537 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.606585 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.606628 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260927 04:03:27.608870 25956 raft_consensus.cc:1261] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Refusing update from remote peer 019aecf88ef147a38cb64fd9cdd7f6a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.609066 25816 raft_consensus.cc:1261] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Refusing update from remote peer 019aecf88ef147a38cb64fd9cdd7f6a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:27.609266 26177 consensus_queue.cc:1035] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e78038c5f7da45b799eaf1502bda9ce0" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.609422 26252 consensus_queue.cc:1035] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b1fd54392a4c8a9a187e1751d5d8d8" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:27.617903 26226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.615335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:40689 (local address 127.17.184.194:37235)
0927 04:03:27.615400 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.615413 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:27.615468 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:27.615631 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:27.615631 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:27.615633 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 04:03:27.615670 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.615671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.616485 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.616486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.617181 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:27.617183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.617529 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.617530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:27.617581 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.617587 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.617742 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.617745 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:27.617797 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 04:03:27.617838 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 04:03:27.617900 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.615393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:37235 (local address 127.17.184.195:40689)
0927 04:03:27.615473 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:27.615475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:27.615534 (+ 59us) server_negotiation.cc:407] Connection header received
0927 04:03:27.615569 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:27.615570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:27.615579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:27.615630 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:27.615712 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.616397 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.617257 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:27.617462 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:27.617533 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:27.617647 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:27.617684 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:27.617742 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:27.617802 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 04:03:27.617823 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:03:27.624667 25659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:27.624776 25659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:27.625414 25659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d80af4dfbc85409b9aa6d5940f464344: Sending DeleteTablet for 3 replicas of tablet 94c33cc627a142b9903e04bb7f5b6bfe
I20260927 04:03:27.625689 26076 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c33cc627a142b9903e04bb7f5b6bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:37818
I20260927 04:03:27.625788 26194 tablet_replica.cc:330] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: stopping tablet replica
I20260927 04:03:27.625834 26194 raft_consensus.cc:2229] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:27.625905 26194 raft_consensus.cc:2258] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.626060 25936 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c33cc627a142b9903e04bb7f5b6bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:59352
I20260927 04:03:27.626138 26196 tablet_replica.cc:330] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: stopping tablet replica
I20260927 04:03:27.626189 26196 raft_consensus.cc:2229] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:27.626225 26196 raft_consensus.cc:2258] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.626487 26194 ts_tablet_manager.cc:1905] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.628106 26196 ts_tablet_manager.cc:1905] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.628185 26194 ts_tablet_manager.cc:1918] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:27.628242 26194 log.cc:1196] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/94c33cc627a142b9903e04bb7f5b6bfe
I20260927 04:03:27.628489 26194 ts_tablet_manager.cc:1939] T 94c33cc627a142b9903e04bb7f5b6bfe P 019aecf88ef147a38cb64fd9cdd7f6a3: Deleting consensus metadata
I20260927 04:03:27.629540 26196 ts_tablet_manager.cc:1918] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 04:03:27.629599 26196 log.cc:1196] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/94c33cc627a142b9903e04bb7f5b6bfe
I20260927 04:03:27.629801 26196 ts_tablet_manager.cc:1939] T 94c33cc627a142b9903e04bb7f5b6bfe P e78038c5f7da45b799eaf1502bda9ce0: Deleting consensus metadata
I20260927 04:03:27.630021 25646 catalog_manager.cc:4841] TS 019aecf88ef147a38cb64fd9cdd7f6a3 (127.17.184.195:40689): tablet 94c33cc627a142b9903e04bb7f5b6bfe (table test-table [id=0ccdceb69293404d9e327daca106d280]) successfully deleted
I20260927 04:03:27.630303 25796 tablet_service.cc:1514] Processing DeleteTablet for tablet 94c33cc627a142b9903e04bb7f5b6bfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:27 UTC) from {username='oryx', principal='oryx/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41360
I20260927 04:03:27.630448 26195 tablet_replica.cc:330] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8: stopping tablet replica
I20260927 04:03:27.630532 26195 raft_consensus.cc:2229] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:27.630591 26195 raft_consensus.cc:2258] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:27.630921 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 25731
I20260927 04:03:27.630949 26195 ts_tablet_manager.cc:1905] T 94c33cc627a142b9903e04bb7f5b6bfe P f9b1fd54392a4c8a9a187e1751d5d8d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:27.636793 25644 catalog_manager.cc:4841] TS e78038c5f7da45b799eaf1502bda9ce0 (127.17.184.194:41359): tablet 94c33cc627a142b9903e04bb7f5b6bfe (table test-table [id=0ccdceb69293404d9e327daca106d280]) successfully deleted
W20260927 04:03:27.641073 25644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:45591 (error 108)
I20260927 04:03:27.641804 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 25871
I20260927 04:03:27.644158 26262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:27.643946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:45591 (local address 127.0.0.1:41378)
0927 04:03:27.644059 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:27.644106 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:45591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":54,"threads_started":1}
W20260927 04:03:27.644271 25644 catalog_manager.cc:4584] TS f9b1fd54392a4c8a9a187e1751d5d8d8 (127.17.184.193:45591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94c33cc627a142b9903e04bb7f5b6bfe: Network error: Client connection negotiation failed: client connection to 127.17.184.193:45591: connect: Connection refused (error 111)
I20260927 04:03:27.649250 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 26011
I20260927 04:03:27.653888 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 25627
2026-09-27T04:03:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2684 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:27 dist-test-slave-zqrn krb5kdc[26265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:27 dist-test-slave-zqrn krb5kdc[26265](info): set up 2 sockets
Sep 27 04:03:27 dist-test-slave-zqrn krb5kdc[26265](info): commencing operation
krb5kdc: starting...
W20260927 04:03:28.764321 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:28 dist-test-slave-zqrn krb5kdc[26265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:28.886833 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:46875
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:46875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:28.975102 26278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:28.975348 26278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:28.975389 26278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:28.976573 26278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:28.976641 26278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:28.976668 26278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:28.976693 26278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:28.976737 26278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:28.978267 26278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:46875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:46875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:28.978559 26278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:28.978791 26278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:28.984088 26284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:28.984077 26286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:28.984169 26278 server_base.cc:1034] running on GCE node
W20260927 04:03:28.984077 26283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:28.984542 26278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:28.984836 26278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:28.985988 26278 hybrid_clock.cc:648] HybridClock initialized: now 1790481808985966 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T04:03:29Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1340 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:29 dist-test-slave-zqrn krb5kdc[26291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:29 dist-test-slave-zqrn krb5kdc[26291](info): set up 2 sockets
Sep 27 04:03:29 dist-test-slave-zqrn krb5kdc[26291](info): commencing operation
krb5kdc: starting...
W20260927 04:03:30.048346 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:30.183168 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33741
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:33741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:30.273445 26304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:30.273746 26304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:30.273788 26304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:30.275075 26304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:30.275156 26304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:30.275179 26304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:30.275198 26304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:30.275225 26304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:30.276793 26304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:33741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:30.277077 26304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:30.277323 26304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:30.282488 26310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:30.282488 26312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:30.282732 26309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:30.282625 26304 server_base.cc:1034] running on GCE node
I20260927 04:03:30.283118 26304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:30.283416 26304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:30.284572 26304 hybrid_clock.cc:648] HybridClock initialized: now 1790481810284554 us; error 34 us; skew 500 ppm
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:30.298024 26304 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:30.298678 26304 webserver.cc:460] Webserver started at http://127.17.184.254:41765/ using document root <none> and password file <none>
I20260927 04:03:30.298970 26304 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:30.299022 26304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:30.299121 26304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:30.300236 26304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "2c4fe0927285489db842e2aca92140c5"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "4b846979fc7ffaedceb1fa5b619bc2b3"
server_key_iv: "3c9d26992a5cfc662b7e059d71bd92b3"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.300542 26304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "2c4fe0927285489db842e2aca92140c5"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "4b846979fc7ffaedceb1fa5b619bc2b3"
server_key_iv: "3c9d26992a5cfc662b7e059d71bd92b3"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.301855 26304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:30.302738 26319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.302991 26304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:30.303153 26304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "2c4fe0927285489db842e2aca92140c5"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "4b846979fc7ffaedceb1fa5b619bc2b3"
server_key_iv: "3c9d26992a5cfc662b7e059d71bd92b3"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.303241 26304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:30.331955 26304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:30.334411 26304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:30.334616 26304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:30.339179 26304 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:33741
I20260927 04:03:30.339248 26371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:33741 every 8 connection(s)
I20260927 04:03:30.339658 26304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:30.340605 26372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:30.343281 26372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Bootstrap starting.
I20260927 04:03:30.343997 26372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:30.344326 26372 log.cc:824] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:30.345556 26372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: No bootstrap required, opened a new log
I20260927 04:03:30.348163 26372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } }
I20260927 04:03:30.348353 26372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:30.348378 26372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4fe0927285489db842e2aca92140c5, State: Initialized, Role: FOLLOWER
I20260927 04:03:30.348551 26372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } }
I20260927 04:03:30.348637 26372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:30.348699 26372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:30.348752 26372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:30.349107 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 26304
I20260927 04:03:30.349272 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:30.349460 18147 external_mini_cluster.cc:1428] Setting key 61ae4353d655d0c7e49bd0714bb1e899
I20260927 04:03:30.349603 26372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } }
I20260927 04:03:30.349725 26372 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4fe0927285489db842e2aca92140c5; no voters:
I20260927 04:03:30.349956 26372 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:30.349987 26375 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:30.350113 26375 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4fe0927285489db842e2aca92140c5, State: Running, Role: LEADER
I20260927 04:03:30.350461 26372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:30.350200 26375 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } }
I20260927 04:03:30.350951 26377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c4fe0927285489db842e2aca92140c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } } }
I20260927 04:03:30.351066 26377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:30.351240 26378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4fe0927285489db842e2aca92140c5. Latest consensus state: current_term: 1 leader_uuid: "2c4fe0927285489db842e2aca92140c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } } }
I20260927 04:03:30.351320 26378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:30.351451 26381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:30.352349 26381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:30.354004 26381 catalog_manager.cc:1347] Generated new cluster ID: cf52db398e194210b64c9026510cf27c
I20260927 04:03:30.354063 26381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:30.363721 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:30.350150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.17.184.254:33741)
0927 04:03:30.350527 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:30.350539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:30.350580 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:03:30.350801 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:30.350810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:30.350871 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:30.351025 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:30.352400 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.354154 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.355397 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.356090 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.356211 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:30.359364 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:30.359370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:30.359371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:30.359389 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:30.361754 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:30.362201 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:30.362204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:30.362205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:30.362342 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:30.362554 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:30.362555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:30.362555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:30.362949 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:30.363128 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:30.363313 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 04:03:30.363464 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":148,"threads_started":1}
I20260927 04:03:30.381140 26381 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:30.381726 26381 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:30.388202 26381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Generated new TSK 0
I20260927 04:03:30.388428 26381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:03:30.419834 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33741
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:30.505736 26398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:30.505963 26398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:30.506033 26398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:30.507507 26398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:30.507565 26398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:30.507596 26398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:30.509178 26398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:30.509421 26398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:30.509639 26398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:30.515057 26406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:30.515066 26404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:30.515066 26403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:30.515179 26398 server_base.cc:1034] running on GCE node
I20260927 04:03:30.515470 26398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:30.515687 26398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:30.516834 26398 hybrid_clock.cc:648] HybridClock initialized: now 1790481810516818 us; error 30 us; skew 500 ppm
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:30.519726 26398 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:03:30.520244 26398 webserver.cc:460] Webserver started at http://127.17.184.193:45819/ using document root <none> and password file <none>
I20260927 04:03:30.520527 26398 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:30.520586 26398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:30.520740 26398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:30.521955 26398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "6090966c00c34457903227996c81e949"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "48cfa249cf0a13d04fa1af3c757ee394"
server_key_iv: "a032193e5039b49d0b24183dfbe589b4"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.522351 26398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "6090966c00c34457903227996c81e949"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "48cfa249cf0a13d04fa1af3c757ee394"
server_key_iv: "a032193e5039b49d0b24183dfbe589b4"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.523958 26398 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:30.524982 26413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.525328 26398 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:30.525463 26398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "6090966c00c34457903227996c81e949"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "48cfa249cf0a13d04fa1af3c757ee394"
server_key_iv: "a032193e5039b49d0b24183dfbe589b4"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.525548 26398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:30.552270 26398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:30.554801 26398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:30.555004 26398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:30.555289 26398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:30.555631 26398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:30.555683 26398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.555720 26398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:30.555747 26398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.561942 26398 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:41219
I20260927 04:03:30.561990 26526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:41219 every 8 connection(s)
I20260927 04:03:30.562335 26398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:30.565402 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 26398
I20260927 04:03:30.565490 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:30.565675 18147 external_mini_cluster.cc:1428] Setting key 62e58863e52039fa658b85165f54c9be
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:30.573040 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:30.563146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:45877 (local address 127.17.184.254:33741)
0927 04:03:30.563229 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:30.563232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:30.563411 (+ 179us) server_negotiation.cc:407] Connection header received
0927 04:03:30.563904 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:30.563905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:30.563921 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:30.563979 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:30.564934 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.565491 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.566852 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.567128 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.567209 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:30.569891 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:30.569893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:30.569893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:30.569900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:30.571412 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:30.571952 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:30.571952 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:30.571953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:30.572005 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:30.572300 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:30.572301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:30.572301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:30.572419 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:30.572473 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:30.572887 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 04:03:30.572971 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 04:03:30.573330 26529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:30.563093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:45877)
0927 04:03:30.563284 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:30.563324 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:30.563810 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:30.564032 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:30.564035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:30.564637 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:30.564838 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:30.564843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.565589 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.565590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:30.566752 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:30.566755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.567217 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.567218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:30.567355 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:30.567955 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:30.567968 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:30.569788 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:30.571496 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:30.571506 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:30.571508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:30.571870 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:30.572072 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:30.572073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:30.572074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:30.572231 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:30.572487 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:30.572493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:30.572672 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 04:03:30.572803 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 04:03:30.572996 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
I20260927 04:03:30.573907 26527 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33741
I20260927 04:03:30.574021 26527 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:30.574218 26527 heartbeater.cc:511] Master 127.17.184.254:33741 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:03:30.574828 26336 ts_manager.cc:194] Registered new tserver with Master: 6090966c00c34457903227996c81e949 (127.17.184.193:41219)
I20260927 04:03:30.575495 26336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:45877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:03:30.595093 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33741
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:30.678233 26532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:30.678462 26532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:30.678488 26532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:30.679824 26532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:30.679875 26532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:30.679905 26532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:30.681403 26532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:30.681669 26532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:30.681910 26532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:30.687076 26537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:30.687080 26540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:30.687175 26538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:30.687367 26532 server_base.cc:1034] running on GCE node
I20260927 04:03:30.687732 26532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:30.688000 26532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:30.689182 26532 hybrid_clock.cc:648] HybridClock initialized: now 1790481810689163 us; error 31 us; skew 500 ppm
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:30.691990 26532 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:03:30.692461 26532 webserver.cc:460] Webserver started at http://127.17.184.194:35185/ using document root <none> and password file <none>
I20260927 04:03:30.692696 26532 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:30.692761 26532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:30.692906 26532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:30.693995 26532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "184ffd8e0c574b908bbcc41943e77ff5"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "51dd0bd6cb940a4a67a75ae373ef80c6"
server_key_iv: "cf4534af2a542a127094476a44c1a2c4"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.694343 26532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "184ffd8e0c574b908bbcc41943e77ff5"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "51dd0bd6cb940a4a67a75ae373ef80c6"
server_key_iv: "cf4534af2a542a127094476a44c1a2c4"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.695799 26532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:30.696518 26547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.696700 26532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:30.696796 26532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "184ffd8e0c574b908bbcc41943e77ff5"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "51dd0bd6cb940a4a67a75ae373ef80c6"
server_key_iv: "cf4534af2a542a127094476a44c1a2c4"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.696866 26532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:30.718570 26532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:30.721036 26532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:30.721215 26532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:30.721460 26532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:30.721794 26532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:30.721848 26532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.721889 26532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:30.721917 26532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.727857 26532 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:43897
I20260927 04:03:30.727895 26660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:43897 every 8 connection(s)
I20260927 04:03:30.728237 26532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:30.730124 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 26532
I20260927 04:03:30.730209 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:30.730387 18147 external_mini_cluster.cc:1428] Setting key 7bf721fce1be20604d8d70c959c5aaec
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
I20260927 04:03:30.739090 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:30.729049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:42771 (local address 127.17.184.254:33741)
0927 04:03:30.729110 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:30.729112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:30.729362 (+ 250us) server_negotiation.cc:407] Connection header received
0927 04:03:30.729935 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:30.729936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:30.729949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:30.729996 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:30.731096 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.731595 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.732637 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.732837 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.732933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:30.735606 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:30.735609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:30.735609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:30.735617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:30.737199 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:30.737755 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:30.737757 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:30.737757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:30.737827 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:30.738204 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:30.738205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:30.738205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:30.738336 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:30.738402 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:30.738811 (+ 409us) server_negotiation.cc:299] Negotiation successful
0927 04:03:30.738966 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:03:30.739257 26663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:30.729034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:42771)
0927 04:03:30.729214 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:30.729251 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:30.729816 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:30.730069 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:30.730073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:30.730742 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:30.731005 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:30.731011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.731691 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.731692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:30.732529 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:30.732531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.732942 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.732943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:30.733078 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:30.733634 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:30.733642 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:30.735423 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:30.737293 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:30.737304 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:30.737307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:30.737653 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:30.737908 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:30.737909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:30.737909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:30.738080 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:30.738417 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:30.738425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:30.738641 (+ 216us) client_negotiation.cc:769] Sending connection context
0927 04:03:30.738769 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 04:03:30.738984 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260927 04:03:30.739804 26661 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33741
I20260927 04:03:30.739980 26661 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:30.740195 26661 heartbeater.cc:511] Master 127.17.184.254:33741 requested a full tablet report, sending...
I20260927 04:03:30.740593 26336 ts_manager.cc:194] Registered new tserver with Master: 184ffd8e0c574b908bbcc41943e77ff5 (127.17.184.194:43897)
I20260927 04:03:30.741076 26336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:42771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:03:30.759550 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33741
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:30.843477 26666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:30.843684 26666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:30.843708 26666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:30.845014 26666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:30.845062 26666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:30.845090 26666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:30.846661 26666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:30.846971 26666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:30.847432 26666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:30.853448 26674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:30.853528 26672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:30.853571 26666 server_base.cc:1034] running on GCE node
W20260927 04:03:30.853448 26671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:30.853997 26666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:30.854323 26666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:30.855473 26666 hybrid_clock.cc:648] HybridClock initialized: now 1790481810855461 us; error 28 us; skew 500 ppm
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:30.858067 26666 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:03:30.858675 26666 webserver.cc:460] Webserver started at http://127.17.184.195:44879/ using document root <none> and password file <none>
I20260927 04:03:30.858969 26666 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:30.859052 26666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:30.859189 26666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:30.860325 26666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "81309bd741c94ac0ab18f985259705da"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "510f9de4f4791923dd696150f79b246c"
server_key_iv: "ca1e0b8bf6bdf59f65f0af53923ebe05"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.860688 26666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "81309bd741c94ac0ab18f985259705da"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "510f9de4f4791923dd696150f79b246c"
server_key_iv: "ca1e0b8bf6bdf59f65f0af53923ebe05"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.862075 26666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:30.862798 26681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.863018 26666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:30.863116 26666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "81309bd741c94ac0ab18f985259705da"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "510f9de4f4791923dd696150f79b246c"
server_key_iv: "ca1e0b8bf6bdf59f65f0af53923ebe05"
server_key_version: "encryptionkey@0"
I20260927 04:03:30.863194 26666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:30.892984 26666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:30.895217 26666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:30.895386 26666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:30.895625 26666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:30.895948 26666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:30.895999 26666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.896041 26666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:30.896068 26666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:30.901944 26666 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:45083
I20260927 04:03:30.901986 26794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:45083 every 8 connection(s)
I20260927 04:03:30.902340 26666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:30.904914 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 26666
I20260927 04:03:30.905006 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:30.905177 18147 external_mini_cluster.cc:1428] Setting key 7b25b7cede533309f7434b7addb10e46
Sep 27 04:03:30 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481810, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:30.912144 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:30.903114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:46637 (local address 127.17.184.254:33741)
0927 04:03:30.903190 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:30.903193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:30.903404 (+ 211us) server_negotiation.cc:407] Connection header received
0927 04:03:30.903872 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:30.903874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:30.903884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:30.903937 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:30.904904 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.905521 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.906738 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.906978 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.907060 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:30.909357 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:30.909358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:30.909359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:30.909365 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:30.910615 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:30.911099 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:30.911100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:30.911100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:30.911158 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:30.911434 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:30.911435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:30.911436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:30.911648 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:30.911722 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:30.911991 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 04:03:30.912062 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:03:30.912324 26797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:30.903094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:46637)
0927 04:03:30.903284 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:30.903321 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:30.903790 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:30.903988 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:30.903991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:30.904560 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:30.904814 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:30.904822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.905601 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.905602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:30.906613 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:30.906616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:30.907069 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:30.907070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:30.907203 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:30.907744 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:30.907752 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:30.909249 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:30.910691 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:30.910697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:30.910698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:30.911019 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:30.911213 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:30.911214 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:30.911214 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:30.911317 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:30.911735 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:30.911740 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:30.911872 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 04:03:30.911963 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 04:03:30.912090 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":102,"threads_started":1}
I20260927 04:03:30.912861 26795 heartbeater.cc:347] Connected to a master server at 127.17.184.254:33741
I20260927 04:03:30.913012 26795 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:30.913249 26795 heartbeater.cc:511] Master 127.17.184.254:33741 requested a full tablet report, sending...
I20260927 04:03:30.913791 26336 ts_manager.cc:194] Registered new tserver with Master: 81309bd741c94ac0ab18f985259705da (127.17.184.195:45083)
I20260927 04:03:30.914186 26336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:46637
I20260927 04:03:30.916877 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab.
Entry for principal oryx/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab.
WARNING: no policy specified for oryx/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab.
Entry for principal oryx/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab.
WARNING: no policy specified for oryx/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab.
Entry for principal oryx/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab.
I20260927 04:03:31.003136 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 26398
I20260927 04:03:31.008460 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 26532
I20260927 04:03:31.013279 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 26666
I20260927 04:03:31.018522 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 26304
I20260927 04:03:31.024116 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33741
--webserver_interface=127.17.184.254
--webserver_port=41765
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:33741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:31.107194 26804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:31.107406 26804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:31.107429 26804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:31.108546 26804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:31.108582 26804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:31.108597 26804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:31.108610 26804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:31.108623 26804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:31.110069 26804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:33741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=41765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:31.110270 26804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:31.110438 26804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:31.115434 26810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:31.115434 26812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:31.115478 26809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:31.115787 26804 server_base.cc:1034] running on GCE node
I20260927 04:03:31.116096 26804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:31.116377 26804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:31.117519 26804 hybrid_clock.cc:648] HybridClock initialized: now 1790481811117511 us; error 37 us; skew 500 ppm
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481811, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:31.120368 26804 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:31.120843 26804 webserver.cc:460] Webserver started at http://127.17.184.254:41765/ using document root <none> and password file <none>
I20260927 04:03:31.121057 26804 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:31.121134 26804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:31.122622 26804 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:03:31.123430 26819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.123631 26804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:03:31.123787 26804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "2c4fe0927285489db842e2aca92140c5"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "4b846979fc7ffaedceb1fa5b619bc2b3"
server_key_iv: "3c9d26992a5cfc662b7e059d71bd92b3"
server_key_version: "encryptionkey@0"
I20260927 04:03:31.124140 26804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:31.131691 26804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:31.133728 26804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:31.133852 26804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:31.137974 26804 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:33741
I20260927 04:03:31.138002 26871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:33741 every 8 connection(s)
I20260927 04:03:31.138417 26804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:31.139113 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 26804
I20260927 04:03:31.139421 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:41219
--local_ip_for_outbound_sockets=127.17.184.193
--tserver_master_addrs=127.17.184.254:33741
--webserver_port=45819
--webserver_interface=127.17.184.193
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.184.193
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:03:31.140120 26872 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:03:31.140796 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Bootstrap starting.
I20260927 04:03:31.150377 26872 log.cc:824] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:31.152403 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:03:31.152737 26872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Bootstrap complete.
I20260927 04:03:31.155179 26872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } }
I20260927 04:03:31.155413 26872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4fe0927285489db842e2aca92140c5, State: Initialized, Role: FOLLOWER
I20260927 04:03:31.155581 26872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } }
I20260927 04:03:31.155664 26872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:31.155694 26872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:31.155754 26872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:03:31.156826 26872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } }
I20260927 04:03:31.156936 26872 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4fe0927285489db842e2aca92140c5; no voters:
I20260927 04:03:31.157142 26872 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 04:03:31.157191 26876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 04:03:31.157327 26876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [term 2 LEADER]: Becoming Leader. State: Replica: 2c4fe0927285489db842e2aca92140c5, State: Running, Role: LEADER
I20260927 04:03:31.157414 26872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:31.157419 26876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } }
I20260927 04:03:31.157811 26878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2c4fe0927285489db842e2aca92140c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } } }
I20260927 04:03:31.157819 26877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4fe0927285489db842e2aca92140c5. Latest consensus state: current_term: 2 leader_uuid: "2c4fe0927285489db842e2aca92140c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4fe0927285489db842e2aca92140c5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33741 } } }
I20260927 04:03:31.157989 26878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:31.158010 26877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:31.158629 26885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:31.159162 26885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:31.159260 26885 catalog_manager.cc:1259] Loaded cluster ID: cf52db398e194210b64c9026510cf27c
I20260927 04:03:31.159283 26885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:31.160017 26885 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:31.160195 26885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2c4fe0927285489db842e2aca92140c5: Loaded TSK: 0
I20260927 04:03:31.160425 26885 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 04:03:31.225134 26874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:31.225384 26874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:31.225428 26874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:31.226684 26874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:31.226765 26874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:31.226809 26874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:31.228430 26874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.193.keytab
--principal=oryx/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:41219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=45819
--tserver_master_addrs=127.17.184.254:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:31.228682 26874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:31.228935 26874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:31.234155 26899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:31.234138 26898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:31.234130 26901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:31.234738 26874 server_base.cc:1034] running on GCE node
I20260927 04:03:31.234992 26874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:31.235297 26874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:31.236486 26874 hybrid_clock.cc:648] HybridClock initialized: now 1790481811236468 us; error 35 us; skew 500 ppm
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481811, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:31.239715 26874 init.cc:377] Logged in from keytab as oryx/127.17.184.193@KRBTEST.COM (short username oryx)
I20260927 04:03:31.240232 26874 webserver.cc:460] Webserver started at http://127.17.184.193:45819/ using document root <none> and password file <none>
I20260927 04:03:31.240474 26874 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:31.240554 26874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:31.242064 26874 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:31.242795 26908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.242993 26874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:31.243091 26874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "6090966c00c34457903227996c81e949"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "48cfa249cf0a13d04fa1af3c757ee394"
server_key_iv: "a032193e5039b49d0b24183dfbe589b4"
server_key_version: "encryptionkey@0"
I20260927 04:03:31.243477 26874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:31.261193 26874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:31.263556 26874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:31.263724 26874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:31.263969 26874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:31.264288 26874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:31.264339 26874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.264384 26874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:31.264410 26874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.270964 26874 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:41219
I20260927 04:03:31.271010 27021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:41219 every 8 connection(s)
I20260927 04:03:31.271365 26874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:31.274767 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 26874
I20260927 04:03:31.275185 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:43897
--local_ip_for_outbound_sockets=127.17.184.194
--tserver_master_addrs=127.17.184.254:33741
--webserver_port=35185
--webserver_interface=127.17.184.194
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.184.194
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:31.280966 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.272221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:54797)
0927 04:03:31.272415 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.272464 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.272935 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.273384 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.273388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.274227 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.274432 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.274438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.276167 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.276169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.277103 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.277106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.277768 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.277769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.277922 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.278484 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.278493 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.280728 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
W20260927 04:03:31.281227 27022 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:33741 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.184.254:33741: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
W20260927 04:03:31.281426 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.272321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:54797 (local address 127.17.184.254:33741)
0927 04:03:31.272504 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.272514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.272556 (+ 42us) server_negotiation.cc:407] Connection header received
0927 04:03:31.273137 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.273144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.273200 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.273338 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.274574 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.276041 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.277227 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.277672 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.277807 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.280880 (+ 3073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:54797: BlockingRecv error: recv got EOF from 127.17.184.193:54797 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:31.285995 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.281430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:54009)
0927 04:03:31.281802 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.281812 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.281884 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.282025 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.282025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.282154 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.282197 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.282199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.282826 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.282827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.283396 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.283397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.283767 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.283768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.283831 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.284270 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.284275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.285905 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":98,"threads_started":1}
W20260927 04:03:31.286048 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.281615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:54009 (local address 127.17.184.254:33741)
0927 04:03:31.281682 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.281684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.281869 (+ 185us) server_negotiation.cc:407] Connection header received
0927 04:03:31.281948 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.281948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.281963 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.282018 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.282278 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.282744 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.283489 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.283677 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.283768 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.285981 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:54009: BlockingRecv error: recv got EOF from 127.17.184.193:54009 (error 108)
Metrics: {"server-negotiator.queue_time_us":48,"thread_start_us":90,"threads_started":1}
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:31.290730 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.286510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:55031)
0927 04:03:31.286554 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.286565 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.286602 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.286796 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.286797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.286978 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.287014 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.287015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.287634 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.287635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.288123 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.288125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.288492 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.288493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.288555 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.288994 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.288999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.290645 (+ 1646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 04:03:31.290783 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.286574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:55031 (local address 127.17.184.254:33741)
0927 04:03:31.286684 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.286686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.286695 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:31.286714 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.286714 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.286723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.286778 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.287085 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.287545 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.288221 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.288414 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.288491 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.290720 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:55031: BlockingRecv error: recv got EOF from 127.17.184.193:55031 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 04:03:31.291015 27022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:03:31.363996 27026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:31.364315 27026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:31.364360 27026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:31.365945 27026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:31.365989 27026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:31.366024 27026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:31.367476 27026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.194.keytab
--principal=oryx/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:43897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=35185
--tserver_master_addrs=127.17.184.254:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:31.367663 27026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:31.367830 27026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:31.373418 27033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:31.373437 27026 server_base.cc:1034] running on GCE node
W20260927 04:03:31.373473 27034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:31.373456 27036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:31.373778 27026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:31.374051 27026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:31.375291 27026 hybrid_clock.cc:648] HybridClock initialized: now 1790481811375278 us; error 30 us; skew 500 ppm
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481811, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:31.377913 27026 init.cc:377] Logged in from keytab as oryx/127.17.184.194@KRBTEST.COM (short username oryx)
I20260927 04:03:31.378410 27026 webserver.cc:460] Webserver started at http://127.17.184.194:35185/ using document root <none> and password file <none>
I20260927 04:03:31.378647 27026 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:31.378705 27026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:31.380187 27026 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:31.380901 27043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.381114 27026 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:31.381197 27026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "184ffd8e0c574b908bbcc41943e77ff5"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "51dd0bd6cb940a4a67a75ae373ef80c6"
server_key_iv: "cf4534af2a542a127094476a44c1a2c4"
server_key_version: "encryptionkey@0"
I20260927 04:03:31.381554 27026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:31.405990 27026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:31.408386 27026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:31.408514 27026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:31.408824 27026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:31.409150 27026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:31.409195 27026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.409219 27026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:31.409232 27026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.415221 27026 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:43897
I20260927 04:03:31.415275 27156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:43897 every 8 connection(s)
I20260927 04:03:31.415666 27026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:31.421224 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 27026
I20260927 04:03:31.421548 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:45083
--local_ip_for_outbound_sockets=127.17.184.195
--tserver_master_addrs=127.17.184.254:33741
--webserver_port=44879
--webserver_interface=127.17.184.195
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.184.195
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 04:03:31.423635 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.416492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:52393 (local address 127.17.184.254:33741)
0927 04:03:31.416576 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.416578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.416787 (+ 209us) server_negotiation.cc:407] Connection header received
0927 04:03:31.417308 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.417309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.417324 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.417374 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.418347 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.418855 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.420014 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.420234 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.420307 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.423577 (+ 3270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:52393: BlockingRecv error: recv got EOF from 127.17.184.194:52393 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:03:31.423702 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.416458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:52393)
0927 04:03:31.416641 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.416679 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.417219 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.417436 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.417439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.418044 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.418249 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.418255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.419032 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.419033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.419900 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.419903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.420327 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.420328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.420459 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.421108 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.421117 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.423492 (+ 2375us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
W20260927 04:03:31.423978 27157 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:33741 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.184.254:33741: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:31.429117 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.424232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:60935)
0927 04:03:31.424314 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.424325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.424363 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.424510 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.424510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.424623 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.424669 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.424671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.425608 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.425608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.426111 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.426113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.426598 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.426599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.426647 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.427200 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.427204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.429067 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 04:03:31.429256 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.424303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:60935 (local address 127.17.184.254:33741)
0927 04:03:31.424362 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.424365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.424374 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:31.424425 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.424426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.424438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.424500 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.424734 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.425527 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.426192 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.426525 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.426603 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.429198 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:60935: BlockingRecv error: recv got EOF from 127.17.184.194:60935 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:31.433557 27159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.429389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:50691)
0927 04:03:31.429425 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.429449 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.429498 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.429640 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.429641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.429764 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.429788 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.429789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.430389 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.430390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.430886 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.430887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.431278 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.431279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.431332 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.431777 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.431781 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.433471 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":21}
W20260927 04:03:31.433609 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.429420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:50691 (local address 127.17.184.254:33741)
0927 04:03:31.429475 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.429477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.429486 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:31.429577 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.429578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.429588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.429639 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.429864 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.430316 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.431017 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.431201 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.431268 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.433544 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:50691: BlockingRecv error: recv got EOF from 127.17.184.194:50691 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260927 04:03:31.433698 27157 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:03:31.506337 27160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:31.506547 27160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:31.506582 27160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:31.507827 27160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:31.507874 27160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:31.507901 27160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:31.509403 27160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/krb5kdc/oryx_127.17.184.195.keytab
--principal=oryx/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:45083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=44879
--tserver_master_addrs=127.17.184.254:33741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:31.509666 27160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:31.509907 27160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:31.515187 27165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:31.515194 27168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:31.515208 27166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:31.515237 27160 server_base.cc:1034] running on GCE node
I20260927 04:03:31.515614 27160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:31.515924 27160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:31.517074 27160 hybrid_clock.cc:648] HybridClock initialized: now 1790481811517060 us; error 32 us; skew 500 ppm
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481811, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:31.519677 27160 init.cc:377] Logged in from keytab as oryx/127.17.184.195@KRBTEST.COM (short username oryx)
I20260927 04:03:31.520108 27160 webserver.cc:460] Webserver started at http://127.17.184.195:44879/ using document root <none> and password file <none>
I20260927 04:03:31.520308 27160 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:31.520361 27160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:31.521718 27160 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:31.522246 27175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.522418 27160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:03:31.522482 27160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "81309bd741c94ac0ab18f985259705da"
format_stamp: "Formatted at 2026-09-27 04:03:30 on dist-test-slave-zqrn"
server_key: "510f9de4f4791923dd696150f79b246c"
server_key_iv: "ca1e0b8bf6bdf59f65f0af53923ebe05"
server_key_version: "encryptionkey@0"
I20260927 04:03:31.522809 27160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:31.534480 27160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:31.536741 27160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:31.536855 27160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:31.537032 27160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:31.537343 27160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:31.537379 27160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.537402 27160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:31.537446 27160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:31.543454 27160 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:45083
I20260927 04:03:31.543496 27288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:45083 every 8 connection(s)
I20260927 04:03:31.543872 27160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:31.547420 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 27160
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 04:03:31.552249 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.544577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:43721 (local address 127.17.184.254:33741)
0927 04:03:31.544662 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.544664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.544910 (+ 246us) server_negotiation.cc:407] Connection header received
0927 04:03:31.545403 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.545404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.545415 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.545545 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.546544 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.547150 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.548416 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.548694 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.548759 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.552201 (+ 3442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:43721: BlockingRecv error: recv got EOF from 127.17.184.195:43721 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 04:03:31.552371 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.544551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:43721)
0927 04:03:31.544790 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.544827 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.545293 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.545629 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.545633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.546245 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.546440 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.546445 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.547269 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.547270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.548310 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.548312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.548825 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.548826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.548934 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.549456 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.549465 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.552117 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":172,"threads_started":1}
W20260927 04:03:31.552626 27289 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:33741 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.184.254:33741: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
I20260927 04:03:31.556886 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.548001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43698 (local address 127.17.184.254:33741)
0927 04:03:31.548073 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.548075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.548204 (+ 129us) server_negotiation.cc:407] Connection header received
0927 04:03:31.548263 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.548264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.548276 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.548358 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.549118 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.549603 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.550627 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.550831 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.550929 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.552403 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:31.552408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:31.552408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:31.552421 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:31.555397 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:31.555810 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:31.555814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:31.555814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:31.555942 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:31.556168 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:31.556169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:31.556169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:31.556404 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:31.556549 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:31.556679 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 04:03:31.556795 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:31.558846 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.552912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:39559)
0927 04:03:31.552942 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.552952 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.552993 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.553998 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.554001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.554204 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.554311 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.554315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.555133 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.555135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.555920 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.555923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.556287 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.556289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.556366 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.556824 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.556831 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.558777 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13}
W20260927 04:03:31.558854 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.553691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:39559 (local address 127.17.184.254:33741)
0927 04:03:31.553796 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.553798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.553806 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:31.553828 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.553829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.553844 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.553896 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.554493 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.555042 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.556003 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.556182 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.556285 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.558805 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:39559: BlockingRecv error: recv got EOF from 127.17.184.195:39559 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:31 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:31.563349 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:31.559146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:43781)
0927 04:03:31.559194 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:31.559205 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:31.559250 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:31.559434 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:31.559435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:31.559558 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:31.559586 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.559587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.560224 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.560225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.560715 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:31.560716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.561072 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.561073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:31.561125 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.561525 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:31.561530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:31.563266 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:03:31.563402 27025 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:31.559215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:43781 (local address 127.17.184.254:33741)
0927 04:03:31.559254 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:31.559255 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:31.559263 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:31.559340 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:31.559341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:31.559349 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:31.559427 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:31.559673 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.560146 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.560800 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:31.560994 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:31.561071 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:31.563330 (+ 2259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:43781: BlockingRecv error: recv got EOF from 127.17.184.195:43781 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 04:03:31.563527 27289 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 04:03:32 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:32 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:32.296376 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:32.291458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:51643)
0927 04:03:32.291601 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:32.291617 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:32.291673 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:32.291835 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:32.291836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:32.291996 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:32.292058 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:32.292060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.292777 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.292778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:32.293338 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:32.293340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.293703 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.293704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:32.293745 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:32.294246 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:32.294251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:32.296309 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260927 04:03:32.296478 27294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:32.291548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:51643 (local address 127.17.184.254:33741)
0927 04:03:32.291696 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:32.291698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:32.291706 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:32.291733 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:32.291734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:32.291743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:32.291815 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:32.292126 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.292700 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.293422 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.293636 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.293695 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:32.296400 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:51643: BlockingRecv error: recv got EOF from 127.17.184.193:51643 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
Sep 27 04:03:32 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:32 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:32.438946 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:32.434110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:41505)
0927 04:03:32.434395 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:32.434408 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:32.434485 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:32.434694 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:32.434695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:32.434863 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:32.434943 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:32.434944 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.435626 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.435627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:32.436196 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:32.436198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.436559 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.436560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:32.436616 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:32.437057 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:32.437061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:32.438804 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":81,"threads_started":1}
W20260927 04:03:32.438982 27294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:32.434212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:41505 (local address 127.17.184.254:33741)
0927 04:03:32.434393 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:32.434395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:32.434464 (+ 69us) server_negotiation.cc:407] Connection header received
0927 04:03:32.434544 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:32.434545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:32.434561 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:32.434629 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:32.435025 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.435535 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.436280 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.436468 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.436560 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:32.438885 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:41505: BlockingRecv error: recv got EOF from 127.17.184.194:41505 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 27 04:03:32 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:32 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:32.569326 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:32.564033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:33665)
0927 04:03:32.564215 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:32.564229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:32.564306 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:32.564480 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:32.564481 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:32.564677 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:32.564728 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:32.564730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.565467 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.565468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:32.566123 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:32.566125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.566531 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.566532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:32.566594 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:32.567332 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:32.567337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:32.569209 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":103,"threads_started":1}
W20260927 04:03:32.569362 27294 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:32.564126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:33665 (local address 127.17.184.254:33741)
0927 04:03:32.564206 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:32.564208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:32.564285 (+ 77us) server_negotiation.cc:407] Connection header received
0927 04:03:32.564370 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:32.564371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:32.564387 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:32.564459 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:32.564811 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.565376 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.566230 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:32.566442 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:32.566530 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:32.569291 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:33665: BlockingRecv error: recv got EOF from 127.17.184.195:33665 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 04:03:33 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:33 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:33.301852 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:33.297038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:51427)
0927 04:03:33.297252 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:33.297268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:33.297364 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:33.297504 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:33.297505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:33.297715 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:33.297767 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:33.297769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.298443 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.298444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:33.299097 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:33.299100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.299436 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.299436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:33.299481 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:33.299956 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:33.299961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:33.301788 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
W20260927 04:03:33.301930 27298 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:33.297246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:51427 (local address 127.17.184.254:33741)
0927 04:03:33.297388 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:33.297390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:33.297398 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:33.297426 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:33.297426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:33.297438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:33.297499 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:33.297846 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.298360 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.299175 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.299368 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.299424 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:33.301877 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:51427: BlockingRecv error: recv got EOF from 127.17.184.193:51427 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
Sep 27 04:03:33 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:33 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:33.444310 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:33.439511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:36283)
0927 04:03:33.439765 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:33.439803 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:33.439864 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:33.440007 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:33.440008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:33.440178 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:33.440231 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:33.440232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.440862 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.440862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:33.441442 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:33.441443 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.441806 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.441807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:33.441855 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:33.442367 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:33.442372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:33.444249 (+ 1877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":187,"threads_started":1}
W20260927 04:03:33.444480 27298 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:33.439610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:36283 (local address 127.17.184.254:33741)
0927 04:03:33.439671 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:33.439673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:33.439861 (+ 188us) server_negotiation.cc:407] Connection header received
0927 04:03:33.439931 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:33.439931 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:33.439945 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:33.439991 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:33.440296 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.440787 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.441522 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.441710 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.441778 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:33.444372 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:36283: BlockingRecv error: recv got EOF from 127.17.184.194:36283 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 27 04:03:33 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:33 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:33.575186 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:33.569899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:41575)
0927 04:03:33.570084 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:33.570098 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:33.570164 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:33.570341 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:33.570342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:33.570521 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:33.570579 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:33.570582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.571440 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.571441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:33.572031 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:33.572033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.572436 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.572437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:33.572484 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:33.573019 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:33.573024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:33.575126 (+ 2102us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":78,"threads_started":1}
W20260927 04:03:33.575335 27298 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:33.570060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:41575 (local address 127.17.184.254:33741)
0927 04:03:33.570190 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:33.570192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:33.570200 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:33.570224 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:33.570225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:33.570238 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:33.570296 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:33.570710 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.571282 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.572133 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:33.572359 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:33.572428 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:33.575241 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:41575: BlockingRecv error: recv got EOF from 127.17.184.195:41575 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 04:03:34 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:34 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:34.307070 27301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:34.302450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:52743)
0927 04:03:34.302611 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:34.302625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:34.302689 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:34.302862 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:34.302862 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:34.303046 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:34.303111 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:34.303113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.303757 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.303758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:34.304326 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:34.304328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.304648 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.304649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:34.304689 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:34.305118 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:34.305122 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:34.306972 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
W20260927 04:03:34.307152 27302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:34.302596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:52743 (local address 127.17.184.254:33741)
0927 04:03:34.302730 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:34.302732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:34.302743 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:34.302772 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:34.302773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:34.302786 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:34.302847 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:34.303178 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.303685 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.304395 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.304582 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.304634 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:34.307070 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:52743: BlockingRecv error: recv got EOF from 127.17.184.193:52743 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
Sep 27 04:03:34 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:34 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:34.449942 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:34.444902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:40703)
0927 04:03:34.445036 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:34.445048 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:34.445111 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:34.445322 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:34.445322 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:34.445481 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:34.445532 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:34.445533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.446326 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.446328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:34.446942 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:34.446945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.447322 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.447323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:34.447383 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:34.447833 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:34.447838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:34.449842 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
W20260927 04:03:34.449985 27302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:34.444973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:40703 (local address 127.17.184.254:33741)
0927 04:03:34.445146 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:34.445148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:34.445158 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:34.445188 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:34.445188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:34.445200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:34.445301 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:34.445704 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.446241 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.447031 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.447234 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.447321 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:34.449922 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:40703: BlockingRecv error: recv got EOF from 127.17.184.194:40703 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 27 04:03:34 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:34 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:34.580327 27304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:34.575762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:51615)
0927 04:03:34.575908 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:34.575920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:34.575990 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:34.576166 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:34.576167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:34.576335 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:34.576383 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:34.576385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.577033 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.577034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:34.577546 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:34.577547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.577928 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.577929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:34.577994 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:34.578507 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:34.578512 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:34.580212 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
W20260927 04:03:34.580354 27302 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:34.575835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:51615 (local address 127.17.184.254:33741)
0927 04:03:34.575905 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:34.575907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:34.575985 (+ 78us) server_negotiation.cc:407] Connection header received
0927 04:03:34.576046 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:34.576047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:34.576059 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:34.576117 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:34.576451 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.576962 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.577644 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:34.577844 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:34.577927 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:34.580293 (+ 2366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:51615: BlockingRecv error: recv got EOF from 127.17.184.195:51615 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 04:03:35 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:35 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:35.317294 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:35.307651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:44725)
0927 04:03:35.307811 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:35.307824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:35.307900 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:35.308096 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:35.308098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:35.308304 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:35.308355 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:35.308357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.309092 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.309092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:35.309633 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:35.309634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.310040 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.310041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:35.310083 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:35.315068 (+ 4985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:35.315074 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:35.317230 (+ 2156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
W20260927 04:03:35.317492 27306 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:35.307705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:44725 (local address 127.17.184.254:33741)
0927 04:03:35.307831 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:35.307833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:35.307883 (+ 50us) server_negotiation.cc:407] Connection header received
0927 04:03:35.307958 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:35.307959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:35.307974 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:35.308055 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:35.308481 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.309003 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.309759 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.309965 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.310029 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:35.317378 (+ 7349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:44725: BlockingRecv error: recv got EOF from 127.17.184.193:44725 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
Sep 27 04:03:35 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:35 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:35.456193 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:35.450641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:51349)
0927 04:03:35.450809 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:35.450842 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:35.450933 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:35.451136 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:35.451137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:35.451342 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:35.451394 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:35.451396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.452079 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.452080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:35.452670 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:35.452671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.453051 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.453052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:35.453122 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:35.453646 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:35.453652 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:35.456097 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
W20260927 04:03:35.456290 27306 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:35.450787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:51349 (local address 127.17.184.254:33741)
0927 04:03:35.450871 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:35.450873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:35.450888 (+ 15us) server_negotiation.cc:407] Connection header received
0927 04:03:35.451023 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:35.451024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:35.451044 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:35.451128 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:35.451479 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.451983 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.452760 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.452962 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.453051 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:35.456219 (+ 3168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:51349: BlockingRecv error: recv got EOF from 127.17.184.194:51349 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 04:03:35 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:35 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:35.586465 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:35.580965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:36335)
0927 04:03:35.581140 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:35.581150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:35.581206 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:35.581341 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:35.581342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:35.581502 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:35.581551 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:35.581553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.582304 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.582305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:35.582967 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:35.582969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.583384 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.583386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:35.583442 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:35.584012 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:35.584017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:35.586402 (+ 2385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
W20260927 04:03:35.586725 27306 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:35.581106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:36335 (local address 127.17.184.254:33741)
0927 04:03:35.581151 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:35.581153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:35.581193 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:03:35.581251 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:35.581252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:35.581264 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:35.581328 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:35.581649 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.582171 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.583065 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:35.583272 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:35.583340 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:35.586587 (+ 3247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:36335: BlockingRecv error: recv got EOF from 127.17.184.195:36335 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 27 04:03:36 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:36 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:36.323218 27309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:36.317881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:42011)
0927 04:03:36.318061 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:36.318075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:36.318146 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:36.318311 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:36.318312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:36.318486 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:36.318538 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:36.318540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.319308 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.319309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:36.319886 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:36.319889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.320324 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.320325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:36.320381 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:36.320969 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:36.320975 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:36.323133 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
W20260927 04:03:36.323347 27310 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:36.317982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:42011 (local address 127.17.184.254:33741)
0927 04:03:36.318109 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:36.318111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:36.318127 (+ 16us) server_negotiation.cc:407] Connection header received
0927 04:03:36.318223 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:36.318224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:36.318234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:36.318286 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:36.318615 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.319222 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.319997 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.320229 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.320286 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:36.323269 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:42011: BlockingRecv error: recv got EOF from 127.17.184.193:42011 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
Sep 27 04:03:36 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:36 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:36.462036 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:36.456749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:52207)
0927 04:03:36.456921 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:36.456933 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:36.456994 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:36.457195 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:36.457196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:36.457434 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:36.457495 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:36.457496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.458347 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.458348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:36.458972 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:36.458974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.459326 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.459327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:36.459368 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:36.459909 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:36.459914 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:36.461974 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
W20260927 04:03:36.462186 27310 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:36.456940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:52207 (local address 127.17.184.254:33741)
0927 04:03:36.457011 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:36.457012 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:36.457021 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:36.457060 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:36.457060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:36.457070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:36.457121 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:36.457590 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.458262 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.459051 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.459258 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.459317 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:36.462081 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:52207: BlockingRecv error: recv got EOF from 127.17.184.194:52207 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 04:03:36 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:36 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:36.591789 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:36.587144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:41807)
0927 04:03:36.587293 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:36.587307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:36.587378 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:36.587531 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:36.587532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:36.587699 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:36.587753 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:36.587755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.588435 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.588436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:36.588970 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:36.588976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.589331 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.589333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:36.589385 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:36.589820 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:36.589825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:36.591721 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
W20260927 04:03:36.591885 27310 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:36.587254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:41807 (local address 127.17.184.254:33741)
0927 04:03:36.587299 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:36.587301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:36.587373 (+ 72us) server_negotiation.cc:407] Connection header received
0927 04:03:36.587454 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:36.587454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:36.587465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:36.587514 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:36.587839 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.588341 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.589053 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:36.589248 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:36.589312 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:36.591816 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:41807: BlockingRecv error: recv got EOF from 127.17.184.195:41807 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 04:03:37 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:37 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:37.329519 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:37.323954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:38103)
0927 04:03:37.324061 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:37.324075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:37.324151 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:37.324397 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:37.324398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:37.324608 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:37.324666 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:37.324667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.325536 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.325537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:37.326105 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:37.326107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.326502 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.326503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:37.326556 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:37.327236 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:37.327240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:37.329456 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
W20260927 04:03:37.329645 27313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:37.323924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:38103 (local address 127.17.184.254:33741)
0927 04:03:37.324061 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:37.324063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:37.324254 (+ 191us) server_negotiation.cc:407] Connection header received
0927 04:03:37.324283 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:37.324284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:37.324298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:37.324409 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:37.324767 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.325359 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.326213 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.326407 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.326468 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:37.329564 (+ 3096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:38103: BlockingRecv error: recv got EOF from 127.17.184.193:38103 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
Sep 27 04:03:37 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:37 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:37.467665 27315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:37.462672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:41491)
0927 04:03:37.462874 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:37.462887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:37.462974 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:37.463128 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:37.463128 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:37.463328 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:37.463375 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:37.463377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.464050 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.464050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:37.464585 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:37.464587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.464950 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.464951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:37.464996 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:37.465486 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:37.465492 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:37.467606 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
W20260927 04:03:37.467799 27313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:37.462816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:41491 (local address 127.17.184.254:33741)
0927 04:03:37.462945 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:37.462947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:37.462962 (+ 15us) server_negotiation.cc:407] Connection header received
0927 04:03:37.463042 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:37.463043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:37.463054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:37.463108 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:37.463459 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.463973 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.464671 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.464876 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.464936 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:37.467713 (+ 2777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:41491: BlockingRecv error: recv got EOF from 127.17.184.194:41491 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 27 04:03:37 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:37 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:37.597242 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:37.592370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:51399)
0927 04:03:37.592549 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:37.592560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:37.592626 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:37.592767 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:37.592768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:37.592928 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:37.592976 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:37.592977 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.593665 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.593666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:37.594236 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:37.594238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.594629 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.594630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:37.594685 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:37.595309 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:37.595315 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:37.597178 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
W20260927 04:03:37.597463 27313 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:37.592481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:51399 (local address 127.17.184.254:33741)
0927 04:03:37.592561 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:37.592562 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:37.592621 (+ 59us) server_negotiation.cc:407] Connection header received
0927 04:03:37.592691 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:37.592692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:37.592706 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:37.592751 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:37.593040 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.593556 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.594335 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:37.594532 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:37.594600 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:37.597391 (+ 2791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:51399: BlockingRecv error: recv got EOF from 127.17.184.195:51399 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 04:03:38 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:38 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:38.335287 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:38.330050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:59791)
0927 04:03:38.330213 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:38.330235 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:38.330289 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:38.330432 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:38.330433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:38.330589 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:38.330638 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:38.330640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.331437 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.331438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:38.331992 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:38.331993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.332428 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.332429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:38.332477 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:38.332996 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:38.333001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:38.335227 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":123,"threads_started":1}
W20260927 04:03:38.335487 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:38.330164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:59791 (local address 127.17.184.254:33741)
0927 04:03:38.330297 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:38.330299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:38.330308 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:38.330340 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:38.330341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:38.330352 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:38.330418 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:38.330730 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.331355 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.332089 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.332333 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.332396 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:38.335394 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:59791: BlockingRecv error: recv got EOF from 127.17.184.193:59791 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
Sep 27 04:03:38 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:38 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:38.474162 27319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:38.468252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:53693)
0927 04:03:38.468413 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:38.468426 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:38.468486 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:38.468780 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:38.468781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:38.468950 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:38.469004 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:38.469005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.470012 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.470014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:38.470853 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:38.470857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.471359 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.471360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:38.471412 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:38.472029 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:38.472034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:38.474100 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
W20260927 04:03:38.474383 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:38.468399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:53693 (local address 127.17.184.254:33741)
0927 04:03:38.468489 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:38.468491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:38.468503 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:03:38.468565 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:38.468566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:38.468577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:38.468629 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:38.469119 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.469898 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.471045 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.471272 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.471336 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:38.474283 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:53693: BlockingRecv error: recv got EOF from 127.17.184.194:53693 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 04:03:38 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:38 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:38.602624 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:38.597834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:50313)
0927 04:03:38.597999 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:38.598011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:38.598088 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:38.598253 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:38.598254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:38.598417 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:38.598466 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:38.598468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.599204 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.599205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:38.599752 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:38.599754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.600109 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.600110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:38.600153 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:38.600657 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:38.600662 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:38.602526 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
W20260927 04:03:38.602751 27318 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:38.597935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:50313 (local address 127.17.184.254:33741)
0927 04:03:38.598014 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:38.598016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:38.598113 (+ 97us) server_negotiation.cc:407] Connection header received
0927 04:03:38.598151 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:38.598152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:38.598167 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:38.598233 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:38.598547 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.599115 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.599850 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:38.600036 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:38.600100 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:38.602617 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:50313: BlockingRecv error: recv got EOF from 127.17.184.195:50313 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 04:03:39 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:39 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:39.340919 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:39.335744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:59573)
0927 04:03:39.335902 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:39.335921 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:39.335976 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:39.336264 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:39.336265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:39.336436 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:39.336485 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:39.336486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.337236 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.337237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:39.337783 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:39.337784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.338166 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.338167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:39.338228 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:39.338886 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:39.338969 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:39.340847 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":58,"threads_started":1}
W20260927 04:03:39.341070 27322 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:39.335949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:59573 (local address 127.17.184.254:33741)
0927 04:03:39.336099 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:39.336101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:39.336110 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:39.336138 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:39.336139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:39.336150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:39.336225 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:39.336567 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.337126 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.337874 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.338091 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.338163 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:39.340978 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:59573: BlockingRecv error: recv got EOF from 127.17.184.193:59573 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
Sep 27 04:03:39 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:39 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:39.480647 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:39.474847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:56627)
0927 04:03:39.475117 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:39.475130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:39.475238 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:39.475428 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:39.475428 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:39.475611 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:39.475670 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:39.475671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.476483 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.476484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:39.477075 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:39.477077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.477474 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.477475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:39.477530 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:39.478354 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:39.478359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:39.480552 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":158,"threads_started":1}
W20260927 04:03:39.480712 27322 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:39.474990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:56627 (local address 127.17.184.254:33741)
0927 04:03:39.475106 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:39.475108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:39.475219 (+ 111us) server_negotiation.cc:407] Connection header received
0927 04:03:39.475327 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:39.475328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:39.475342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:39.475420 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:39.475756 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.476395 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.477175 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.477398 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.477476 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:39.480648 (+ 3172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:56627: BlockingRecv error: recv got EOF from 127.17.184.194:56627 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 04:03:39 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:39 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:39.608495 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:39.603211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:42063)
0927 04:03:39.603360 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:39.603372 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:39.603426 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:39.603623 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:39.603624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:39.603799 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:39.603849 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:39.603851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.604591 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.604591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:39.605126 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:39.605127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.605773 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.605774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:39.605828 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:39.606383 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:39.606388 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:39.608421 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260927 04:03:39.608721 27322 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:39.603292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:42063 (local address 127.17.184.254:33741)
0927 04:03:39.603407 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:39.603409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:39.603422 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:03:39.603502 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:39.603503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:39.603519 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:39.603602 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:39.603937 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.604512 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.605239 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:39.605446 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:39.605517 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:39.608649 (+ 3132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:42063: BlockingRecv error: recv got EOF from 127.17.184.195:42063 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 04:03:40 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:40 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:40.352105 27325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:40.341480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:52935)
0927 04:03:40.341620 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:40.341639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:40.341690 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:40.341897 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:40.341898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:40.342058 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:40.342109 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:40.342110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.342795 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.342796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:40.343417 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:40.343419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.343834 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.343835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:40.343882 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:40.350095 (+ 6213us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:40.350101 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:40.352044 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
W20260927 04:03:40.352295 27326 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:40.341605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:52935 (local address 127.17.184.254:33741)
0927 04:03:40.341756 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:40.341758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:40.341767 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:40.341791 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:40.341791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:40.341806 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:40.341879 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:40.342172 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.342691 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.343500 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.343746 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.343813 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:40.352226 (+ 8413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:52935: BlockingRecv error: recv got EOF from 127.17.184.193:52935 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
Sep 27 04:03:40 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:40 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:40.486173 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:40.481182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:50239)
0927 04:03:40.481329 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:40.481339 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:40.481400 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:40.481527 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:40.481527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:40.481698 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:40.481745 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:40.481746 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.482474 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.482475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:40.483145 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:40.483148 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.483534 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.483535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:40.483580 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:40.484065 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:40.484070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:40.486101 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
W20260927 04:03:40.486276 27326 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:40.481237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:50239 (local address 127.17.184.254:33741)
0927 04:03:40.481293 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:40.481295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:40.481403 (+ 108us) server_negotiation.cc:407] Connection header received
0927 04:03:40.481453 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:40.481453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:40.481465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:40.481506 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:40.481851 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.482365 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.483239 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.483456 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.483515 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:40.486205 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:50239: BlockingRecv error: recv got EOF from 127.17.184.194:50239 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 27 04:03:40 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:40 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:40.614040 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:40.609096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:49687)
0927 04:03:40.609300 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:40.609311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:40.609374 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:40.609523 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:40.609524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:40.609697 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:40.609746 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:40.609748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.610398 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.610399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:40.611030 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:40.611032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.611344 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.611345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:40.611386 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:40.611941 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:40.611946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:40.613975 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":127,"threads_started":1}
W20260927 04:03:40.614194 27326 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:40.609213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:49687 (local address 127.17.184.254:33741)
0927 04:03:40.609278 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:40.609280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:40.609374 (+ 94us) server_negotiation.cc:407] Connection header received
0927 04:03:40.609439 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:40.609440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:40.609452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:40.609498 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:40.609818 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.610315 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.611098 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:40.611279 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:40.611329 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:40.614101 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:49687: BlockingRecv error: recv got EOF from 127.17.184.195:49687 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 04:03:41 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:41 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:41.357781 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:41.352751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:55303)
0927 04:03:41.352972 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:41.352987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:41.353078 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:41.353328 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:41.353330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:41.353518 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:41.353567 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:41.353569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.354317 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.354318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:41.354877 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:41.354879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.355284 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.355285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:41.355330 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:41.355862 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:41.355868 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:41.357689 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
W20260927 04:03:41.357947 27330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:41.352868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:55303 (local address 127.17.184.254:33741)
0927 04:03:41.353015 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:41.353017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:41.353060 (+ 43us) server_negotiation.cc:407] Connection header received
0927 04:03:41.353153 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:41.353154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:41.353168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:41.353298 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:41.353699 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.354229 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.355013 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.355208 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.355270 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:41.357843 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:55303: BlockingRecv error: recv got EOF from 127.17.184.193:55303 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":173,"threads_started":1}
Sep 27 04:03:41 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:41 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:41.492406 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:41.486846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:45465)
0927 04:03:41.487042 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:41.487057 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:41.487122 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:41.487518 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:41.487519 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:41.487706 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:41.487805 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:41.487807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.488519 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.488520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:41.489179 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:41.489181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.489614 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.489616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:41.489671 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:41.490280 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:41.490285 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:41.492343 (+ 2058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":123,"threads_started":1}
W20260927 04:03:41.492637 27330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:41.487017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:45465 (local address 127.17.184.254:33741)
0927 04:03:41.487105 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:41.487107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:41.487117 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:41.487240 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:41.487241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:41.487252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:41.487340 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:41.487910 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.488417 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.489290 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.489506 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.489581 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:41.492483 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:45465: BlockingRecv error: recv got EOF from 127.17.184.194:45465 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 04:03:41 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:41 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:41.619944 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:41.614603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:46239)
0927 04:03:41.614786 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:41.614802 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:41.614871 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:41.615218 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:41.615219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:41.615401 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:41.615449 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:41.615451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.616270 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.616271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:41.616867 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:41.616869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.617260 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.617261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:41.617309 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:41.617847 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:41.617851 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:41.619874 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":73,"threads_started":1}
W20260927 04:03:41.620039 27330 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:41.614820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:46239 (local address 127.17.184.254:33741)
0927 04:03:41.614983 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:41.614985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:41.615003 (+ 18us) server_negotiation.cc:407] Connection header received
0927 04:03:41.615028 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:41.615029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:41.615043 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:41.615142 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:41.615589 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.616166 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.616968 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:41.617164 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:41.617234 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:41.619970 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:46239: BlockingRecv error: recv got EOF from 127.17.184.195:46239 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Sep 27 04:03:42 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:42 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:42.363389 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:42.358240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:50069)
0927 04:03:42.358393 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:42.358406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:42.358475 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:42.358720 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:42.358721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:42.358948 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:42.359006 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:42.359008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.359749 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.359750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:42.360306 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:42.360308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.360717 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.360718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:42.360775 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:42.361366 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:42.361371 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:42.363316 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260927 04:03:42.363554 27334 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:42.358329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:50069 (local address 127.17.184.254:33741)
0927 04:03:42.358481 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:42.358483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:42.358521 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:03:42.358547 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:42.358548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:42.358585 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:42.358709 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:42.359098 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.359651 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.360405 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.360613 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.360681 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:42.363454 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:50069: BlockingRecv error: recv got EOF from 127.17.184.193:50069 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
Sep 27 04:03:42 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:42 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:42.498974 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:42.493105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:51139)
0927 04:03:42.493272 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:42.493287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:42.493365 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:42.493563 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:42.493564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:42.493771 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:42.493822 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:42.493824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.494716 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.494718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:42.495322 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:42.495324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.495822 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.495823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:42.495911 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:42.496504 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:42.496511 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:42.498837 (+ 2326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
W20260927 04:03:42.499065 27334 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:42.493247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:51139 (local address 127.17.184.254:33741)
0927 04:03:42.493298 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:42.493300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:42.493344 (+ 44us) server_negotiation.cc:407] Connection header received
0927 04:03:42.493468 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:42.493469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:42.493481 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:42.493602 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:42.493932 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.494479 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.495442 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.495705 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.495808 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:42.498997 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:51139: BlockingRecv error: recv got EOF from 127.17.184.194:51139 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 27 04:03:42 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:42 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:42.625797 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:42.620426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:44469)
0927 04:03:42.620606 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:42.620615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:42.620666 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:42.620835 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:42.620836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:42.620992 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:42.621037 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:42.621039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.622103 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.622104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:42.622641 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:42.622642 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.623217 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.623218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:42.623260 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:42.623816 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:42.623821 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:42.625685 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
W20260927 04:03:42.625846 27334 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:42.620539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:44469 (local address 127.17.184.254:33741)
0927 04:03:42.620613 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:42.620615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:42.620660 (+ 45us) server_negotiation.cc:407] Connection header received
0927 04:03:42.620744 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:42.620745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:42.620762 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:42.620823 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:42.621114 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.622020 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.622735 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:42.623128 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:42.623205 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:42.625769 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:44469: BlockingRecv error: recv got EOF from 127.17.184.195:44469 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 04:03:43 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:43 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:43.369012 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:43.363969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:60953)
0927 04:03:43.364249 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:43.364264 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:43.364337 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:43.364636 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:43.364637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:43.364859 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:43.364907 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:43.364909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.365597 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.365598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:43.366134 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:43.366135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.366493 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.366493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:43.366534 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:43.367108 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:43.367113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:43.368912 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":63,"threads_started":1}
W20260927 04:03:43.369073 27338 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:43.364315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:60953 (local address 127.17.184.254:33741)
0927 04:03:43.364492 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:43.364494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:43.364503 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:03:43.364528 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:43.364528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:43.364539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:43.364605 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:43.365019 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.365529 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.366216 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.366425 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.366482 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:43.368998 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:60953: BlockingRecv error: recv got EOF from 127.17.184.193:60953 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
Sep 27 04:03:43 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:43 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:43.505051 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:43.499648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:37555)
0927 04:03:43.499801 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:43.499814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:43.499921 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:43.500173 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:43.500174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:43.500375 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:43.500423 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:43.500424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.501200 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.501202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:43.501862 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:43.501864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.502277 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.502279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:43.502347 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:43.502830 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:43.502836 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:43.504943 (+ 2107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
W20260927 04:03:43.505123 27338 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:43.499828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:37555 (local address 127.17.184.254:33741)
0927 04:03:43.499913 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:43.499915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:43.499929 (+ 14us) server_negotiation.cc:407] Connection header received
0927 04:03:43.499993 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:43.499994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:43.500010 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:43.500155 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:43.500519 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.501094 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.501963 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.502192 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.502280 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:43.505040 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:37555: BlockingRecv error: recv got EOF from 127.17.184.194:37555 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 04:03:43 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:43 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:43.631564 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:43.626291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:59989)
0927 04:03:43.626446 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:43.626457 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:43.626508 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:43.626701 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:43.626702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:43.626879 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:43.626964 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:43.626966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.627799 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.627800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:43.628414 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:43.628416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.628797 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.628798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:43.628847 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:43.629385 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:43.629389 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:43.631505 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
W20260927 04:03:43.631699 27338 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:43.626402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:59989 (local address 127.17.184.254:33741)
0927 04:03:43.626500 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:43.626502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:43.626514 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:03:43.626554 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:43.626555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:43.626566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:43.626717 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:43.627056 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.627674 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.628503 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:43.628709 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:43.628768 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:43.631605 (+ 2837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:59989: BlockingRecv error: recv got EOF from 127.17.184.195:59989 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 27 04:03:44 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:44 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:44.374996 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:44.369643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:59083)
0927 04:03:44.369857 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:44.369869 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:44.369939 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:44.370162 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:44.370163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:44.370357 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:44.370404 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:44.370406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.371329 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.371330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:44.371923 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:44.371925 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.372373 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.372373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:44.372420 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:44.372879 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:44.372884 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:44.374921 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":122,"threads_started":1}
W20260927 04:03:44.375154 27342 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:44.369759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:59083 (local address 127.17.184.254:33741)
0927 04:03:44.369875 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:44.369876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:44.369923 (+ 47us) server_negotiation.cc:407] Connection header received
0927 04:03:44.370021 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:44.370022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:44.370037 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:44.370107 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:44.370501 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.371247 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.372031 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.372278 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.372364 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:44.375057 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:59083: BlockingRecv error: recv got EOF from 127.17.184.193:59083 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
Sep 27 04:03:44 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:44 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:44.510185 27343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:44.505620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:40349)
0927 04:03:44.505777 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:44.505797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:44.505854 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:44.506003 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:44.506004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:44.506201 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:44.506249 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:44.506251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.506878 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.506879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:44.507451 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:44.507453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.507860 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.507861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:44.507907 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:44.508343 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:44.508348 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:44.510126 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":60,"threads_started":1}
W20260927 04:03:44.510322 27342 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:44.505812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:40349 (local address 127.17.184.254:33741)
0927 04:03:44.505882 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:44.505889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:44.505897 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:03:44.505921 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:44.505921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:44.505932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:44.505979 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:44.506317 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.506803 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.507550 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.507786 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.507845 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:44.510237 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:40349: BlockingRecv error: recv got EOF from 127.17.184.194:40349 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 04:03:44 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:44 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:44.636986 27344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:44.632163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:46975)
0927 04:03:44.632391 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:44.632404 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:44.632498 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:44.632655 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:44.632656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:44.632882 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:44.632926 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:44.632928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.633608 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.633609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:44.634165 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:44.634167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.634525 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.634526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:44.634585 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:44.635140 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:44.635150 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:44.636873 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
W20260927 04:03:44.637027 27342 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:44.632271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:46975 (local address 127.17.184.254:33741)
0927 04:03:44.632377 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:44.632379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:44.632478 (+ 99us) server_negotiation.cc:407] Connection header received
0927 04:03:44.632556 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:44.632557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:44.632572 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:44.632649 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:44.633016 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.633523 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.634265 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:44.634454 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:44.634528 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:44.636960 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:46975: BlockingRecv error: recv got EOF from 127.17.184.195:46975 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 27 04:03:45 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:45 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:45.386991 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:45.375608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:52431)
0927 04:03:45.375797 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:45.375810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:45.375872 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:45.376023 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:45.376024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:45.376206 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:45.376263 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:45.376265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.377015 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.377016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:45.377568 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:45.377569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.377967 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.377968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:45.378012 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:45.384619 (+ 6607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:45.384625 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:45.386862 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":85,"threads_started":1}
W20260927 04:03:45.387072 27346 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:45.375739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:52431 (local address 127.17.184.254:33741)
0927 04:03:45.375869 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:45.375871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:45.375881 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:45.375929 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:45.375929 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:45.375941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:45.376000 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:45.376373 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.376930 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.377653 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.377888 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.377948 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:45.386988 (+ 9040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:52431: BlockingRecv error: recv got EOF from 127.17.184.193:52431 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":51,"threads_started":1}
Sep 27 04:03:45 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:45 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:45.516049 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:45.510816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:45335)
0927 04:03:45.510982 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:45.510995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:45.511060 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:45.511289 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:45.511291 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:45.511485 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:45.511532 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:45.511534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.512321 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.512322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:45.512876 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:45.512878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.513336 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.513337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:45.513408 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:45.513948 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:45.513953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:45.515988 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":131,"threads_started":1}
W20260927 04:03:45.516160 27346 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:45.510965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:45335 (local address 127.17.184.254:33741)
0927 04:03:45.511043 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:45.511045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:45.511055 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:45.511143 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:45.511144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:45.511156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:45.511220 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:45.511635 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.512196 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.512988 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.513241 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.513311 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:45.516090 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:45335: BlockingRecv error: recv got EOF from 127.17.184.194:45335 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 27 04:03:45 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:45 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.195@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:45.642436 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:45.637526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.195:34087)
0927 04:03:45.637664 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:45.637677 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:45.637746 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:45.637932 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:45.637934 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:45.638171 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:45.638230 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:45.638233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.638981 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.638983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:45.639530 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:45.639532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.639941 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.639942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:45.639993 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:45.640429 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:45.640434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:45.642376 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260927 04:03:45.642618 27346 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:45.637582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:34087 (local address 127.17.184.254:33741)
0927 04:03:45.637658 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:45.637661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:45.637738 (+ 77us) server_negotiation.cc:407] Connection header received
0927 04:03:45.637823 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:45.637825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:45.637848 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:45.637906 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:45.638318 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.638809 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.639618 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:45.639855 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:45.639912 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:45.642536 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.195:34087: BlockingRecv error: recv got EOF from 127.17.184.195:34087 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 04:03:46 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:46 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.193@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:46.393972 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:46.387661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.193:56263)
0927 04:03:46.387920 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:46.387938 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:46.388002 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:46.388343 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:46.388344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:46.388509 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:46.388560 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:46.388561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.389323 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.389324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:46.389979 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:46.389982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.390463 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.390464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:46.390517 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:46.391129 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:46.391134 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:46.393888 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":72,"threads_started":1}
W20260927 04:03:46.394166 27350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:46.387955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:56263 (local address 127.17.184.254:33741)
0927 04:03:46.388095 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:46.388097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:46.388107 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:46.388141 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:46.388142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:46.388151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:46.388264 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:46.388670 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.389205 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.390104 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.390344 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.390415 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:46.394037 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.193:56263: BlockingRecv error: recv got EOF from 127.17.184.193:56263 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":100,"threads_started":1}
Sep 27 04:03:46 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 04:03:46 dist-test-slave-zqrn krb5kdc[26291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.184.194@KRBTEST.COM for oryx/127.17.184.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 04:03:46.522056 27351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:46.516643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:33741 (local address 127.17.184.194:46671)
0927 04:03:46.516782 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:46.516794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:46.516864 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:46.517024 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:46.517025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:46.517195 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:46.517249 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:46.517251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.517966 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.517967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:46.518505 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:46.518507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.519244 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.519246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:46.519313 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:46.519846 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:46.519850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:46.521993 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.184.254:33741: Server oryx/127.17.184.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
W20260927 04:03:46.522240 27350 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:46.516794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:46671 (local address 127.17.184.254:33741)
0927 04:03:46.516861 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:46.516863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:46.516873 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:46.516909 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:46.516910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:46.516920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:46.516976 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:46.517327 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.517883 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.518609 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.519071 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.519196 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:46.522146 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.184.194:46671: BlockingRecv error: recv got EOF from 127.17.184.194:46671 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:03:46.549263 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 26874
I20260927 04:03:46.554431 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 27026
I20260927 04:03:46.559244 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 27160
I20260927 04:03:46.563788 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 26804
2026-09-27T04:03:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17569 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T04:03:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:46Z Disabled control of system clock
I20260927 04:03:46.593261 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41019
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:41019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 04:03:46.678086 27355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:46.678297 27355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:46.678318 27355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:46.680323 27355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:46.680387 27355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:46.680402 27355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:46.680446 27355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:46.680469 27355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:46.682204 27355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:41019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:46.682461 27355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:46.682753 27355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:46.689111 27361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:46.689246 27360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:46.689164 27355 server_base.cc:1034] running on GCE node
W20260927 04:03:46.689137 27363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:46.689778 27355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:46.690150 27355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:46.691347 27355 hybrid_clock.cc:648] HybridClock initialized: now 1790481826691335 us; error 57 us; skew 500 ppm
I20260927 04:03:46.692817 27355 webserver.cc:460] Webserver started at http://127.17.184.254:36571/ using document root <none> and password file <none>
I20260927 04:03:46.693077 27355 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:46.693148 27355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:46.693305 27355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:46.694443 27355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "1ab68609881f44389f01fca10a52b97f"
format_stamp: "Formatted at 2026-09-27 04:03:46 on dist-test-slave-zqrn"
server_key: "2839524d9fc512c43b90a884ffe6fafa"
server_key_iv: "e271ac659469c9b902b1f64e5e3437d7"
server_key_version: "encryptionkey@0"
I20260927 04:03:46.694816 27355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "1ab68609881f44389f01fca10a52b97f"
format_stamp: "Formatted at 2026-09-27 04:03:46 on dist-test-slave-zqrn"
server_key: "2839524d9fc512c43b90a884ffe6fafa"
server_key_iv: "e271ac659469c9b902b1f64e5e3437d7"
server_key_version: "encryptionkey@0"
I20260927 04:03:46.696229 27355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:46.697158 27369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:46.697366 27355 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:46.697477 27355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "1ab68609881f44389f01fca10a52b97f"
format_stamp: "Formatted at 2026-09-27 04:03:46 on dist-test-slave-zqrn"
server_key: "2839524d9fc512c43b90a884ffe6fafa"
server_key_iv: "e271ac659469c9b902b1f64e5e3437d7"
server_key_version: "encryptionkey@0"
I20260927 04:03:46.697554 27355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:46.712374 27355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:46.712769 27355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:46.713006 27355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:46.717314 27355 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:41019
I20260927 04:03:46.717350 27421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:41019 every 8 connection(s)
I20260927 04:03:46.717734 27355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:46.718276 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 27355
I20260927 04:03:46.718384 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:46.718598 18147 external_mini_cluster.cc:1428] Setting key 02137867b5ef38ee11ba82aed5ccd0d0
I20260927 04:03:46.718675 27422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:46.722500 27422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f: Bootstrap starting.
I20260927 04:03:46.723284 27422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:46.723611 27422 log.cc:824] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:46.724603 27422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f: No bootstrap required, opened a new log
I20260927 04:03:46.724898 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:46.719370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.17.184.254:41019)
0927 04:03:46.719591 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:46.719600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:46.719626 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:03:46.720334 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:46.720341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:46.720406 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:46.720525 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:46.720819 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.721729 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.722833 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.723361 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.723494 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:46.724177 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:46.724183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:46.724183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:46.724362 (+ 179us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:46.724395 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:46.724406 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:46.724489 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:46.724565 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 04:03:46.724649 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260927 04:03:46.726698 27422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab68609881f44389f01fca10a52b97f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41019 } }
I20260927 04:03:46.726876 27422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:46.726965 27422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab68609881f44389f01fca10a52b97f, State: Initialized, Role: FOLLOWER
I20260927 04:03:46.727090 27422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab68609881f44389f01fca10a52b97f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41019 } }
I20260927 04:03:46.727182 27422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:46.727205 27422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:46.727254 27422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:46.727833 27422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab68609881f44389f01fca10a52b97f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41019 } }
I20260927 04:03:46.727952 27422 leader_election.cc:302] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ab68609881f44389f01fca10a52b97f; no voters:
I20260927 04:03:46.728143 27422 leader_election.cc:288] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:46.728176 27427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:46.728343 27427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [term 1 LEADER]: Becoming Leader. State: Replica: 1ab68609881f44389f01fca10a52b97f, State: Running, Role: LEADER
I20260927 04:03:46.728428 27427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab68609881f44389f01fca10a52b97f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41019 } }
I20260927 04:03:46.728467 27422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:46.728785 27428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ab68609881f44389f01fca10a52b97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab68609881f44389f01fca10a52b97f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41019 } } }
I20260927 04:03:46.728799 27427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ab68609881f44389f01fca10a52b97f. Latest consensus state: current_term: 1 leader_uuid: "1ab68609881f44389f01fca10a52b97f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab68609881f44389f01fca10a52b97f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41019 } } }
I20260927 04:03:46.728971 27428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:46.728987 27427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f [sys.catalog]: This master's current role is: LEADER
W20260927 04:03:46.730270 27443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:03:46.730355 27443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:03:46.730433 27435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:46.730553 27435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:46.732235 27435 catalog_manager.cc:1347] Generated new cluster ID: 97372f9214d0408cb504eee030d2715e
I20260927 04:03:46.732333 27435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:46.747673 27435 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:46.748284 27435 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:46.756042 27435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ab68609881f44389f01fca10a52b97f: Generated new TSK 0
I20260927 04:03:46.756269 27435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:03:46.765069 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41019
--builtin_ntp_servers=127.17.184.212:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 04:03:46.866375 27446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:46.866602 27446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:46.866626 27446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:46.867863 27446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:46.867904 27446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:46.867930 27446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:46.869254 27446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:46.869513 27446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:46.869825 27446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:46.875022 27454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:46.874993 27452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:46.875000 27451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:46.875451 27446 server_base.cc:1034] running on GCE node
I20260927 04:03:46.875716 27446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:46.875981 27446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:46.877188 27446 hybrid_clock.cc:648] HybridClock initialized: now 1790481826877144 us; error 57 us; skew 500 ppm
I20260927 04:03:46.878620 27446 webserver.cc:460] Webserver started at http://127.17.184.193:36955/ using document root <none> and password file <none>
I20260927 04:03:46.878934 27446 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:46.879006 27446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:46.879172 27446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:46.880323 27446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "ddd12c5c81ca4100919c18734bbd8d1a"
format_stamp: "Formatted at 2026-09-27 04:03:46 on dist-test-slave-zqrn"
server_key: "44dc11d6a8dd6b9ab4062e05418a46bd"
server_key_iv: "0347373ef83a968b765e2d69266af481"
server_key_version: "encryptionkey@0"
I20260927 04:03:46.880698 27446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "ddd12c5c81ca4100919c18734bbd8d1a"
format_stamp: "Formatted at 2026-09-27 04:03:46 on dist-test-slave-zqrn"
server_key: "44dc11d6a8dd6b9ab4062e05418a46bd"
server_key_iv: "0347373ef83a968b765e2d69266af481"
server_key_version: "encryptionkey@0"
I20260927 04:03:46.882208 27446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:46.883105 27460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:46.883294 27446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:46.883381 27446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "ddd12c5c81ca4100919c18734bbd8d1a"
format_stamp: "Formatted at 2026-09-27 04:03:46 on dist-test-slave-zqrn"
server_key: "44dc11d6a8dd6b9ab4062e05418a46bd"
server_key_iv: "0347373ef83a968b765e2d69266af481"
server_key_version: "encryptionkey@0"
I20260927 04:03:46.883441 27446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:46.901376 27446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:46.901671 27446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:46.901809 27446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:46.902042 27446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:46.902400 27446 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:46.902434 27446 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:46.902459 27446 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:46.902518 27446 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:46.908497 27446 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:40439
I20260927 04:03:46.908603 27573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:40439 every 8 connection(s)
I20260927 04:03:46.909008 27446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:46.913129 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 27446
I20260927 04:03:46.913237 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:46.913493 18147 external_mini_cluster.cc:1428] Setting key 6ef63bfc82f741b09e2c042f6ba06c97
I20260927 04:03:46.914479 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41019
--builtin_ntp_servers=127.17.184.212:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 04:03:46.914819 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:46.909969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:35311 (local address 127.17.184.254:41019)
0927 04:03:46.910032 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:46.910035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:46.910290 (+ 255us) server_negotiation.cc:407] Connection header received
0927 04:03:46.910772 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:46.910773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:46.910806 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:46.910860 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:46.911536 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.912040 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.913235 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.913484 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.913563 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:46.914491 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:46.914493 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:46.914493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:46.914499 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:46.914519 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:46.914524 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:46.914585 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:46.914707 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 04:03:46.914758 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:03:46.915139 27576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:46.909947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41019 (local address 127.17.184.193:35311)
0927 04:03:46.910153 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:46.910191 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:46.910686 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:46.910931 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:46.910934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:46.910943 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:46.911462 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:46.911467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.912137 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.912138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:46.913112 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:46.913115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:46.913570 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:46.913571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:46.913724 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:46.914338 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:46.914339 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:46.914369 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:46.914394 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:46.914618 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:46.914627 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:46.914631 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:46.914701 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:03:46.914821 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260927 04:03:46.915697 27574 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41019
I20260927 04:03:46.915810 27574 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:46.916033 27574 heartbeater.cc:511] Master 127.17.184.254:41019 requested a full tablet report, sending...
I20260927 04:03:46.916625 27386 ts_manager.cc:194] Registered new tserver with Master: ddd12c5c81ca4100919c18734bbd8d1a (127.17.184.193:40439)
I20260927 04:03:46.917420 27386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.193:35311
W20260927 04:03:47.000273 27577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:47.000515 27577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:47.000553 27577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:47.001945 27577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:47.002018 27577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:47.002072 27577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:47.003603 27577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:47.003895 27577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:47.004227 27577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:47.009678 27585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.009811 27583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.009836 27582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:47.009811 27577 server_base.cc:1034] running on GCE node
I20260927 04:03:47.010305 27577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:47.010573 27577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:47.011765 27577 hybrid_clock.cc:648] HybridClock initialized: now 1790481827011734 us; error 48 us; skew 500 ppm
I20260927 04:03:47.013370 27577 webserver.cc:460] Webserver started at http://127.17.184.194:40169/ using document root <none> and password file <none>
I20260927 04:03:47.013649 27577 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:47.013700 27577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:47.013844 27577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:47.015136 27577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "eaf03baaf5bf4282b70c69d03d568a36"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "b8178952669b6579f3276812d0822cd7"
server_key_iv: "b15dc656fe0efe0f85fcf2dcd5e5e32d"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.015527 27577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "eaf03baaf5bf4282b70c69d03d568a36"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "b8178952669b6579f3276812d0822cd7"
server_key_iv: "b15dc656fe0efe0f85fcf2dcd5e5e32d"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.017091 27577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:03:47.018294 27591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.018529 27577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:47.018632 27577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "eaf03baaf5bf4282b70c69d03d568a36"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "b8178952669b6579f3276812d0822cd7"
server_key_iv: "b15dc656fe0efe0f85fcf2dcd5e5e32d"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.018786 27577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:47.049602 27577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:47.049995 27577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:47.050165 27577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:47.050557 27577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:47.051059 27577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:47.051119 27577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.051173 27577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:47.051201 27577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.057612 27577 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:37395
I20260927 04:03:47.057672 27704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:37395 every 8 connection(s)
I20260927 04:03:47.058015 27577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:47.061075 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 27577
I20260927 04:03:47.061159 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:47.061385 18147 external_mini_cluster.cc:1428] Setting key 923da3784cb14f53d90d4238faa806fd
I20260927 04:03:47.062489 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41019
--builtin_ntp_servers=127.17.184.212:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 04:03:47.064148 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.058926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:57183 (local address 127.17.184.254:41019)
0927 04:03:47.058995 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:47.058998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:47.059223 (+ 225us) server_negotiation.cc:407] Connection header received
0927 04:03:47.059741 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:47.059742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:47.059753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:47.059814 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:47.060613 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:47.061203 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:47.062507 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:47.062763 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:47.062850 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:47.063851 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:47.063853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:47.063853 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:47.063858 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:47.063882 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:47.063886 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:47.063950 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:47.064036 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 04:03:47.064066 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 04:03:47.064394 27707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.058844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41019 (local address 127.17.184.194:57183)
0927 04:03:47.059093 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:47.059128 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:47.059622 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:47.059916 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:47.059920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:47.059929 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:47.060479 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:47.060484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:47.061349 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:47.061350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:47.062404 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:47.062407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:47.062936 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:47.062937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:47.063075 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:47.063676 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:47.063677 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:47.063708 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:47.063732 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:47.063953 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:47.063959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:47.063962 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:47.064038 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 04:03:47.064149 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":124,"threads_started":1}
I20260927 04:03:47.064975 27705 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41019
I20260927 04:03:47.065078 27705 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:47.065290 27705 heartbeater.cc:511] Master 127.17.184.254:41019 requested a full tablet report, sending...
I20260927 04:03:47.065739 27386 ts_manager.cc:194] Registered new tserver with Master: eaf03baaf5bf4282b70c69d03d568a36 (127.17.184.194:37395)
I20260927 04:03:47.066154 27386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.194:57183
W20260927 04:03:47.147827 27708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:47.148070 27708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:47.148104 27708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:47.149288 27708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:47.149359 27708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:47.149403 27708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:47.150790 27708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:47.151079 27708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:47.151394 27708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:47.156600 27716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.156600 27713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.156701 27714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:47.157011 27708 server_base.cc:1034] running on GCE node
I20260927 04:03:47.157295 27708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:47.157611 27708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:47.158790 27708 hybrid_clock.cc:648] HybridClock initialized: now 1790481827158763 us; error 40 us; skew 500 ppm
I20260927 04:03:47.160194 27708 webserver.cc:460] Webserver started at http://127.17.184.195:40837/ using document root <none> and password file <none>
I20260927 04:03:47.160488 27708 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:47.160561 27708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:47.160699 27708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:47.161818 27708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "85aa6da246fe4fc8a5df1f0573a7dc42"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "379e2eb1f7afeb54ac05fa6d27c8000d"
server_key_iv: "25db3bdf5767ae8f3e9eeeeb7321da2e"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.162187 27708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "85aa6da246fe4fc8a5df1f0573a7dc42"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "379e2eb1f7afeb54ac05fa6d27c8000d"
server_key_iv: "25db3bdf5767ae8f3e9eeeeb7321da2e"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.163686 27708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:47.164583 27722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.164881 27708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:47.164997 27708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "85aa6da246fe4fc8a5df1f0573a7dc42"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "379e2eb1f7afeb54ac05fa6d27c8000d"
server_key_iv: "25db3bdf5767ae8f3e9eeeeb7321da2e"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.165087 27708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:47.175390 27708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:47.175726 27708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:47.175880 27708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:47.176131 27708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:47.176469 27708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:47.176525 27708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.176569 27708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:47.176606 27708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.183247 27708 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:35333
I20260927 04:03:47.183308 27835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:35333 every 8 connection(s)
I20260927 04:03:47.183709 27708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:47.188354 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 27708
I20260927 04:03:47.188474 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:47.188704 18147 external_mini_cluster.cc:1428] Setting key 1db4049bdd85c17e862fd0470de22a27
I20260927 04:03:47.189989 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.184552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:40843 (local address 127.17.184.254:41019)
0927 04:03:47.184639 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:47.184641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:47.184864 (+ 223us) server_negotiation.cc:407] Connection header received
0927 04:03:47.185412 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:47.185414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:47.185424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:47.185484 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:47.186301 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:47.186883 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:47.188466 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:47.188711 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:47.188812 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:47.189737 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:47.189738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:47.189739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:47.189743 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:47.189761 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:47.189766 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:47.189820 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:47.189905 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 04:03:47.189928 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 04:03:47.190213 27838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.184485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41019 (local address 127.17.184.195:40843)
0927 04:03:47.184704 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:47.184742 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:47.185311 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:47.185601 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:47.185604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:47.185613 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:47.186205 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:47.186210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:47.187031 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:47.187032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:47.188318 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:47.188322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:47.188832 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:47.188833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:47.188978 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:47.189549 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:47.189550 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:47.189581 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:47.189628 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:47.189833 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:47.189837 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:47.189840 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:47.189891 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:03:47.189985 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":118,"threads_started":1}
I20260927 04:03:47.190716 27836 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41019
I20260927 04:03:47.190804 27836 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:47.191080 27836 heartbeater.cc:511] Master 127.17.184.254:41019 requested a full tablet report, sending...
I20260927 04:03:47.191497 27386 ts_manager.cc:194] Registered new tserver with Master: 85aa6da246fe4fc8a5df1f0573a7dc42 (127.17.184.195:35333)
I20260927 04:03:47.191901 27386 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.195:40843
I20260927 04:03:47.201205 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 04:03:47.203550 27425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:47.202794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49980 (local address 127.17.184.254:41019)
0927 04:03:47.202894 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:47.202897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:47.203036 (+ 139us) server_negotiation.cc:407] Connection header received
0927 04:03:47.203178 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:47.203179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:47.203189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:47.203257 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:47.203459 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49980: BlockingRecv error: recv got EOF from 127.0.0.1:49980 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 04:03:47.203855 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 27446
I20260927 04:03:47.209039 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 27577
I20260927 04:03:47.214510 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 27708
I20260927 04:03:47.219451 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 27355
2026-09-27T04:03:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (654 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T04:03:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:47Z Disabled control of system clock
I20260927 04:03:47.247774 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:46251
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 04:03:47.332643 27847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:47.332864 27847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:47.332888 27847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:47.334023 27847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:47.334064 27847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:47.334077 27847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:47.334156 27847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:47.334168 27847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:47.335536 27847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:46251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:47.335752 27847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:47.336004 27847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:47.341463 27855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.341468 27852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.341761 27853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:47.341563 27847 server_base.cc:1034] running on GCE node
I20260927 04:03:47.342032 27847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:47.342368 27847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:47.343565 27847 hybrid_clock.cc:648] HybridClock initialized: now 1790481827343543 us; error 29 us; skew 500 ppm
I20260927 04:03:47.345060 27847 webserver.cc:460] Webserver started at http://127.17.184.254:43571/ using document root <none> and password file <none>
I20260927 04:03:47.345422 27847 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:47.345500 27847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:47.345660 27847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:47.346817 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "ee71fbeb20bf409aa1481ed9ed66dba5"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "72abddf240b4e01d0b71b0532676cdd0"
server_key_iv: "64570b96a801555e1afb7abb6f68209c"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.347272 27847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "ee71fbeb20bf409aa1481ed9ed66dba5"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "72abddf240b4e01d0b71b0532676cdd0"
server_key_iv: "64570b96a801555e1afb7abb6f68209c"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.348760 27847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:47.349653 27861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.350025 27847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:47.350155 27847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "ee71fbeb20bf409aa1481ed9ed66dba5"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "72abddf240b4e01d0b71b0532676cdd0"
server_key_iv: "64570b96a801555e1afb7abb6f68209c"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.350265 27847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:47.354055 27847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:47.354372 27847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:47.354524 27847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:47.358471 27847 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:46251
I20260927 04:03:47.358502 27913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:46251 every 8 connection(s)
I20260927 04:03:47.358846 27847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:47.359566 27914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:47.362386 27914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5: Bootstrap starting.
I20260927 04:03:47.362645 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 27847
I20260927 04:03:47.362751 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:47.362959 18147 external_mini_cluster.cc:1428] Setting key 5881f7d86a9eca37215b9a790c5ce7fa
I20260927 04:03:47.363170 27914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:47.363528 27914 log.cc:824] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:47.364303 27914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5: No bootstrap required, opened a new log
I20260927 04:03:47.365883 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.363546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.17.184.254:46251)
0927 04:03:47.363769 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:47.363782 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:47.363821 (+ 39us) server_negotiation.cc:407] Connection header received
0927 04:03:47.364410 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:47.364421 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:47.364482 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:47.364633 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:47.365382 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:47.365386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:47.365386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:47.365390 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:47.365433 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:47.365444 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:47.365519 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:47.365591 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 04:03:47.365682 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260927 04:03:47.366290 27914 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee71fbeb20bf409aa1481ed9ed66dba5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46251 } }
I20260927 04:03:47.366462 27914 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:47.366499 27914 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee71fbeb20bf409aa1481ed9ed66dba5, State: Initialized, Role: FOLLOWER
I20260927 04:03:47.366631 27914 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee71fbeb20bf409aa1481ed9ed66dba5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46251 } }
I20260927 04:03:47.366714 27914 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:47.366737 27914 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:47.366803 27914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:47.367722 27914 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee71fbeb20bf409aa1481ed9ed66dba5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46251 } }
I20260927 04:03:47.367812 27914 leader_election.cc:302] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee71fbeb20bf409aa1481ed9ed66dba5; no voters:
I20260927 04:03:47.367992 27914 leader_election.cc:288] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:47.368099 27919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:47.368257 27914 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:47.368270 27919 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [term 1 LEADER]: Becoming Leader. State: Replica: ee71fbeb20bf409aa1481ed9ed66dba5, State: Running, Role: LEADER
I20260927 04:03:47.368391 27919 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee71fbeb20bf409aa1481ed9ed66dba5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46251 } }
I20260927 04:03:47.368783 27920 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee71fbeb20bf409aa1481ed9ed66dba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee71fbeb20bf409aa1481ed9ed66dba5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46251 } } }
I20260927 04:03:47.368788 27921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee71fbeb20bf409aa1481ed9ed66dba5. Latest consensus state: current_term: 1 leader_uuid: "ee71fbeb20bf409aa1481ed9ed66dba5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee71fbeb20bf409aa1481ed9ed66dba5" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46251 } } }
I20260927 04:03:47.368947 27920 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:47.368990 27921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5 [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:47.369552 27928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:47.370164 27928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:47.371840 27928 catalog_manager.cc:1347] Generated new cluster ID: d81d7c722b40438f86218b934c391660
I20260927 04:03:47.371894 27928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:47.386801 27928 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:47.387066 27928 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:47.397421 27928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee71fbeb20bf409aa1481ed9ed66dba5: Generated new TSK 0
I20260927 04:03:47.397735 27928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:03:47.404310 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46251
--builtin_ntp_servers=127.17.184.212:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 04:03:47.491232 27938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:47.491458 27938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:47.491485 27938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:47.493032 27938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:47.493085 27938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:47.493114 27938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:47.495029 27938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:47.495378 27938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:47.495738 27938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:47.501228 27944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:47.501428 27938 server_base.cc:1034] running on GCE node
W20260927 04:03:47.501253 27946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.501237 27943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:47.501799 27938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:47.502075 27938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:47.503490 27938 hybrid_clock.cc:648] HybridClock initialized: now 1790481827503461 us; error 42 us; skew 500 ppm
I20260927 04:03:47.505251 27938 webserver.cc:460] Webserver started at http://127.17.184.193:41631/ using document root <none> and password file <none>
I20260927 04:03:47.505548 27938 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:47.505635 27938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:47.505810 27938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:47.507041 27938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "5ee751639bdd4ed9916a635581ec6a58"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "e50e6874061af824887d37f5c7328c95"
server_key_iv: "eeb517a1c4f9f2e3b70de32e9afa7d51"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.507406 27938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "5ee751639bdd4ed9916a635581ec6a58"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "e50e6874061af824887d37f5c7328c95"
server_key_iv: "eeb517a1c4f9f2e3b70de32e9afa7d51"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.509058 27938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:03:47.510028 27952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.510339 27938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:47.510468 27938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "5ee751639bdd4ed9916a635581ec6a58"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "e50e6874061af824887d37f5c7328c95"
server_key_iv: "eeb517a1c4f9f2e3b70de32e9afa7d51"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.510560 27938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:47.514707 27938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:47.515092 27938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:47.515256 27938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:47.515478 27938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:47.515841 27938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:47.515899 27938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.515941 27938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:47.515969 27938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.522354 27938 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:44727
I20260927 04:03:47.522423 28065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:44727 every 8 connection(s)
I20260927 04:03:47.522816 27938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:47.526731 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.524053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:47855 (local address 127.17.184.254:46251)
0927 04:03:47.524188 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:47.524192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:47.524437 (+ 245us) server_negotiation.cc:407] Connection header received
0927 04:03:47.525034 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:47.525035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:47.525043 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:47.525097 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:47.526432 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:47.526433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:47.526433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:47.526439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:47.526463 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:47.526468 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:47.526532 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:47.526636 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 04:03:47.526652 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 04:03:47.527001 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.523913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:46251 (local address 127.17.184.193:47855)
0927 04:03:47.524239 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:47.524335 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:47.524917 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:47.525197 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:47.525201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:47.525210 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:47.526263 (+ 1053us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:47.526264 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:47.526297 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:47.526327 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:47.526544 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:47.526551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:47.526562 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 04:03:47.526648 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:03:47.526714 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260927 04:03:47.527622 28066 heartbeater.cc:347] Connected to a master server at 127.17.184.254:46251
I20260927 04:03:47.527709 28066 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:47.527910 28066 heartbeater.cc:511] Master 127.17.184.254:46251 requested a full tablet report, sending...
I20260927 04:03:47.528439 27878 ts_manager.cc:194] Registered new tserver with Master: 5ee751639bdd4ed9916a635581ec6a58 (127.17.184.193:44727)
I20260927 04:03:47.529834 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 27938
I20260927 04:03:47.529950 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:47.530169 18147 external_mini_cluster.cc:1428] Setting key cf24425e2c30d20ea2571ddfed18a6bf
I20260927 04:03:47.531316 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46251
--builtin_ntp_servers=127.17.184.212:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 04:03:47.619196 28069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:47.619458 28069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:47.619483 28069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:47.620728 28069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:47.620772 28069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:47.620800 28069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:47.622213 28069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:47.622453 28069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:47.622728 28069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:47.629185 28075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.629267 28074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.629249 28077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:47.629384 28069 server_base.cc:1034] running on GCE node
I20260927 04:03:47.629603 28069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:47.629823 28069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:47.631035 28069 hybrid_clock.cc:648] HybridClock initialized: now 1790481827630997 us; error 61 us; skew 500 ppm
I20260927 04:03:47.632447 28069 webserver.cc:460] Webserver started at http://127.17.184.194:46737/ using document root <none> and password file <none>
I20260927 04:03:47.632718 28069 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:47.632768 28069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:47.632891 28069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:47.634074 28069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "46e93f18ea574575ba1b82ac77f4fcf1"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "b484ad192d45ebe14357b3b49bf58096"
server_key_iv: "1b57cefebd2417ea09b26dae5cbd1f4f"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.634460 28069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "46e93f18ea574575ba1b82ac77f4fcf1"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "b484ad192d45ebe14357b3b49bf58096"
server_key_iv: "1b57cefebd2417ea09b26dae5cbd1f4f"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.636089 28069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:03:47.637084 28083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.637301 28069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:47.637388 28069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "46e93f18ea574575ba1b82ac77f4fcf1"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "b484ad192d45ebe14357b3b49bf58096"
server_key_iv: "1b57cefebd2417ea09b26dae5cbd1f4f"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.637454 28069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:47.641067 28069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:47.641320 28069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:47.641415 28069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:47.641580 28069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:47.641840 28069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:47.641873 28069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.641893 28069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:47.641906 28069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.648010 28069 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:36981
I20260927 04:03:47.648051 28196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:36981 every 8 connection(s)
I20260927 04:03:47.648471 28069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:03:47.652725 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.649431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:43471 (local address 127.17.184.254:46251)
0927 04:03:47.649489 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:47.649492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:47.649720 (+ 228us) server_negotiation.cc:407] Connection header received
0927 04:03:47.650400 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:47.650401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:47.650408 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:47.650474 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:47.652197 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:47.652198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:47.652199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:47.652205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:47.652223 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:47.652228 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:47.652289 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:47.652639 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 04:03:47.652657 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:03:47.652817 28199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.649320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:46251 (local address 127.17.184.194:43471)
0927 04:03:47.649534 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:47.649583 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:47.650230 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:47.650598 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:47.650602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:47.650612 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:47.651955 (+ 1343us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:47.651956 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:47.651992 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:47.652027 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:47.652422 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:47.652428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:47.652442 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 04:03:47.652538 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 04:03:47.652606 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260927 04:03:47.653594 28197 heartbeater.cc:347] Connected to a master server at 127.17.184.254:46251
I20260927 04:03:47.653734 28197 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:47.653914 28197 heartbeater.cc:511] Master 127.17.184.254:46251 requested a full tablet report, sending...
I20260927 04:03:47.654536 27878 ts_manager.cc:194] Registered new tserver with Master: 46e93f18ea574575ba1b82ac77f4fcf1 (127.17.184.194:36981)
I20260927 04:03:47.656790 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 28069
I20260927 04:03:47.656879 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:47.657110 18147 external_mini_cluster.cc:1428] Setting key 9eae8733076fc1cb697d999eb1dfaabc
I20260927 04:03:47.658267 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46251
--builtin_ntp_servers=127.17.184.212:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 04:03:47.751140 28200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:47.751389 28200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:47.751416 28200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:47.752837 28200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:47.752884 28200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:47.752915 28200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:47.754490 28200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:47.754781 28200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:47.755250 28200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:47.762477 28206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.762792 28205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:47.763037 28208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:47.763084 28200 server_base.cc:1034] running on GCE node
I20260927 04:03:47.763507 28200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:47.763886 28200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:47.765094 28200 hybrid_clock.cc:648] HybridClock initialized: now 1790481827765069 us; error 42 us; skew 500 ppm
I20260927 04:03:47.767182 28200 webserver.cc:460] Webserver started at http://127.17.184.195:40769/ using document root <none> and password file <none>
I20260927 04:03:47.767599 28200 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:47.767666 28200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:47.767861 28200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:47.769575 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "9e135875ef65453abe522bdcf6fc610b"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "e2ae9a80bce3ee3e664dddc73416390d"
server_key_iv: "393f189ea5a5737677a1cda02eb2357b"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.770170 28200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "9e135875ef65453abe522bdcf6fc610b"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "e2ae9a80bce3ee3e664dddc73416390d"
server_key_iv: "393f189ea5a5737677a1cda02eb2357b"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.771831 28200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260927 04:03:47.772745 28214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.772981 28200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:47.773058 28200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "9e135875ef65453abe522bdcf6fc610b"
format_stamp: "Formatted at 2026-09-27 04:03:47 on dist-test-slave-zqrn"
server_key: "e2ae9a80bce3ee3e664dddc73416390d"
server_key_iv: "393f189ea5a5737677a1cda02eb2357b"
server_key_version: "encryptionkey@0"
I20260927 04:03:47.773144 28200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:47.776829 28200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:47.777127 28200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:47.777256 28200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:47.777485 28200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:47.777815 28200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:47.777848 28200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.777870 28200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:47.777902 28200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:47.784543 28200 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:45443
I20260927 04:03:47.784646 28327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:45443 every 8 connection(s)
I20260927 04:03:47.785002 28200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:47.789099 27917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.786100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:47579 (local address 127.17.184.254:46251)
0927 04:03:47.786171 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:47.786173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:47.786494 (+ 321us) server_negotiation.cc:407] Connection header received
0927 04:03:47.787251 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:47.787253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:47.787259 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:47.787332 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:47.788702 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:47.788703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:47.788703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:03:47.788709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:47.788733 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:03:47.788738 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:47.788804 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:47.788985 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 04:03:47.789001 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 04:03:47.789397 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:47.786022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:46251 (local address 127.17.184.195:47579)
0927 04:03:47.786318 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:47.786364 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:47.787081 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:47.787416 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:47.787420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:47.787447 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:47.788515 (+ 1068us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:03:47.788515 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:47.788550 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:03:47.788584 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:47.788861 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:47.788867 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:47.788870 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:47.788953 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 04:03:47.789010 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":130,"threads_started":1}
I20260927 04:03:47.790009 28328 heartbeater.cc:347] Connected to a master server at 127.17.184.254:46251
I20260927 04:03:47.790139 28328 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:47.790380 28328 heartbeater.cc:511] Master 127.17.184.254:46251 requested a full tablet report, sending...
I20260927 04:03:47.791008 27878 ts_manager.cc:194] Registered new tserver with Master: 9e135875ef65453abe522bdcf6fc610b (127.17.184.195:45443)
I20260927 04:03:47.793921 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 28200
I20260927 04:03:47.794040 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:47.794252 18147 external_mini_cluster.cc:1428] Setting key c884b0aa96c9c4144c67f7ed1e3c1327
I20260927 04:03:47.795403 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 04:03:47.798370 27917 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:03:47.797384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.17.184.254:46251)
0927 04:03:47.797489 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:47.797491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:47.797728 (+ 237us) server_negotiation.cc:407] Connection header received
0927 04:03:47.797752 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:47.797753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:47.797758 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:47.797815 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:47.798180 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42260: BlockingRecv error: recv got EOF from 127.0.0.1:42260 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 04:03:47.798461 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 27938
I20260927 04:03:47.804281 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 28069
I20260927 04:03:47.809278 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 28200
I20260927 04:03:47.813998 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 27847
2026-09-27T04:03:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (594 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:47 dist-test-slave-zqrn krb5kdc[28338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:47 dist-test-slave-zqrn krb5kdc[28338](info): set up 2 sockets
Sep 27 04:03:47 dist-test-slave-zqrn krb5kdc[28338](info): commencing operation
krb5kdc: starting...
W20260927 04:03:48.896700 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:48 dist-test-slave-zqrn krb5kdc[28338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:03:49.027499 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:37071
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:37071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:49.110786 28351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:49.111145 28351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:49.111189 28351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:49.112429 28351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:03:49.112514 28351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:49.112541 28351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:49.112567 28351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:03:49.112598 28351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:03:49.114209 28351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:37071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:37071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:49.114511 28351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:49.114777 28351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:03:49.120199 28351 server_base.cc:1034] running on GCE node
W20260927 04:03:49.120167 28359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:49.120167 28357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:49.120147 28356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:49.120682 28351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:49.120955 28351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:49.122140 28351 hybrid_clock.cc:648] HybridClock initialized: now 1790481829122142 us; error 45 us; skew 500 ppm
Sep 27 04:03:49 dist-test-slave-zqrn krb5kdc[28338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481829, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:49.124965 28351 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:03:49.125352 28351 webserver.cc:460] Webserver started at http://127.17.184.254:43007/ using document root <none> and password file <none>
I20260927 04:03:49.125530 28351 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:49.125568 28351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:49.125662 28351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:49.126654 28351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "c7afce2004d343be8efcc65635b6d79a"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "004d4265ebb7b5c8a3cb8257cd529456"
server_key_iv: "dc6c7036ee20c115dbd5c6e6b69e0e02"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.127018 28351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "c7afce2004d343be8efcc65635b6d79a"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "004d4265ebb7b5c8a3cb8257cd529456"
server_key_iv: "dc6c7036ee20c115dbd5c6e6b69e0e02"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.128299 28351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:49.129098 28366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.129355 28351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:49.129468 28351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "c7afce2004d343be8efcc65635b6d79a"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "004d4265ebb7b5c8a3cb8257cd529456"
server_key_iv: "dc6c7036ee20c115dbd5c6e6b69e0e02"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.129544 28351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:49.144354 28351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:49.146476 28351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:49.146627 28351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:49.150576 28351 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:37071
I20260927 04:03:49.150710 28418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:37071 every 8 connection(s)
I20260927 04:03:49.151037 28351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:03:49.152041 28419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:49.152849 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 28351
I20260927 04:03:49.152931 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:03:49.153132 18147 external_mini_cluster.cc:1428] Setting key 2a67684fc19d9fe289e1a87de778be7c
I20260927 04:03:49.155416 28419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a: Bootstrap starting.
I20260927 04:03:49.156207 28419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:49.156538 28419 log.cc:824] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:49.157337 28419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a: No bootstrap required, opened a new log
I20260927 04:03:49.159431 28419 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7afce2004d343be8efcc65635b6d79a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37071 } }
I20260927 04:03:49.159605 28419 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:49.159682 28419 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7afce2004d343be8efcc65635b6d79a, State: Initialized, Role: FOLLOWER
I20260927 04:03:49.159847 28419 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7afce2004d343be8efcc65635b6d79a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37071 } }
I20260927 04:03:49.159962 28419 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:03:49.160002 28419 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:03:49.160066 28419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 0 FOLLOWER]: Advancing to term 1
Sep 27 04:03:49 dist-test-slave-zqrn krb5kdc[28338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:49.160889 28419 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7afce2004d343be8efcc65635b6d79a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37071 } }
I20260927 04:03:49.160985 28419 leader_election.cc:302] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7afce2004d343be8efcc65635b6d79a; no voters:
I20260927 04:03:49.161245 28419 leader_election.cc:288] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:03:49.161325 28424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:49.161509 28424 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [term 1 LEADER]: Becoming Leader. State: Replica: c7afce2004d343be8efcc65635b6d79a, State: Running, Role: LEADER
I20260927 04:03:49.161543 28419 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:03:49.161608 28424 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7afce2004d343be8efcc65635b6d79a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37071 } }
I20260927 04:03:49.161926 28424 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7afce2004d343be8efcc65635b6d79a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7afce2004d343be8efcc65635b6d79a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37071 } } }
I20260927 04:03:49.162003 28424 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:49.162115 28426 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7afce2004d343be8efcc65635b6d79a. Latest consensus state: current_term: 1 leader_uuid: "c7afce2004d343be8efcc65635b6d79a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7afce2004d343be8efcc65635b6d79a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37071 } } }
I20260927 04:03:49.162236 28429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:03:49.162235 28426 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a [sys.catalog]: This master's current role is: LEADER
I20260927 04:03:49.162796 28429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:03:49.165422 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.153717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.17.184.254:37071)
0927 04:03:49.153940 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.153950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.153981 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:03:49.154565 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.154575 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.154645 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.154801 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:49.155503 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.156899 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.157765 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.158196 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.158337 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.161267 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:49.161272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:49.161272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:49.161286 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:49.163746 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.164165 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.164168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.164168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.164256 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.164471 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.164471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.164472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.164775 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:49.164886 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.165037 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.165153 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260927 04:03:49.167809 28429 catalog_manager.cc:1347] Generated new cluster ID: c1706f1f726a45f999caf0957b178c78
I20260927 04:03:49.167883 28429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:03:49.189778 28429 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:03:49.190375 28429 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:03:49.194497 28429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a: Generated new TSK 0
I20260927 04:03:49.194705 28429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:03:49.230139 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37071
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:49.313122 28445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:49.313390 28445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:49.313426 28445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:49.314620 28445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:49.314695 28445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:49.314742 28445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:03:49.316306 28445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:49.316627 28445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:49.316874 28445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:49.322196 28451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:49.322345 28445 server_base.cc:1034] running on GCE node
W20260927 04:03:49.322220 28453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:49.322196 28450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:49.322626 28445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:49.322837 28445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:49.323979 28445 hybrid_clock.cc:648] HybridClock initialized: now 1790481829323965 us; error 45 us; skew 500 ppm
Sep 27 04:03:49 dist-test-slave-zqrn krb5kdc[28338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481829, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:49.326591 28445 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:03:49.327112 28445 webserver.cc:460] Webserver started at http://127.17.184.193:35611/ using document root <none> and password file <none>
I20260927 04:03:49.327339 28445 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:49.327418 28445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:49.327569 28445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:49.328670 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "204ddd710d444694aa88f5acff0c0207"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "fc60c1a269349d441759ecafb351ad16"
server_key_iv: "f65f4e92d1306cca0349f13f6200afec"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.329047 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "204ddd710d444694aa88f5acff0c0207"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "fc60c1a269349d441759ecafb351ad16"
server_key_iv: "f65f4e92d1306cca0349f13f6200afec"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.330425 28445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:03:49.331254 28460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.331459 28445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:49.331573 28445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "204ddd710d444694aa88f5acff0c0207"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "fc60c1a269349d441759ecafb351ad16"
server_key_iv: "f65f4e92d1306cca0349f13f6200afec"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.331650 28445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:49.346518 28445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:49.348836 28445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:49.349016 28445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:49.349282 28445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:49.349606 28445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:49.349660 28445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.349701 28445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:49.349731 28445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.355330 28445 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:36019
I20260927 04:03:49.355374 28573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:36019 every 8 connection(s)
I20260927 04:03:49.355731 28445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:03:49.355885 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 28445
I20260927 04:03:49.355973 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:03:49.356244 18147 external_mini_cluster.cc:1428] Setting key d64aeb88431eb76e3d73c685997b873c
Sep 27 04:03:49 dist-test-slave-zqrn krb5kdc[28338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481829, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:03:49.365983 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.356549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:51493 (local address 127.17.184.254:37071)
0927 04:03:49.356611 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.356614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.356870 (+ 256us) server_negotiation.cc:407] Connection header received
0927 04:03:49.357418 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.357419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.357432 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.357501 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:49.358503 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.359079 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.360091 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.360390 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.360476 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.363004 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:49.363005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:49.363006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:49.363013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:49.364452 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.364920 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.364921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.364922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.364985 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.365244 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.365245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.365245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.365375 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:49.365454 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.365789 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.365885 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 04:03:49.366225 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.356539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37071 (local address 127.17.184.193:51493)
0927 04:03:49.356706 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.356751 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:49.357307 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:49.357558 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:49.357561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:49.358197 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:49.358394 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.358399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.359171 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.359172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.359984 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.359986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.360487 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.360488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.360618 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.361187 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:49.361195 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:49.362874 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:49.364540 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:49.364547 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:49.364550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:49.364842 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:49.365055 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:49.365056 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:49.365056 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:49.365176 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:49.365444 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:49.365451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:49.365648 (+ 197us) client_negotiation.cc:769] Sending connection context
0927 04:03:49.365773 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 04:03:49.365944 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260927 04:03:49.366693 28574 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37071
I20260927 04:03:49.366801 28574 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:49.367012 28574 heartbeater.cc:511] Master 127.17.184.254:37071 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
I20260927 04:03:49.367599 28383 ts_manager.cc:194] Registered new tserver with Master: 204ddd710d444694aa88f5acff0c0207 (127.17.184.193:36019)
I20260927 04:03:49.368542 28383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:51493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:03:49.387087 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37071
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:49.470754 28579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:49.471041 28579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:49.471078 28579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:49.472276 28579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:49.472368 28579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:49.472414 28579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:03:49.473951 28579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:49.474231 28579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:49.474470 28579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:03:49.479817 28585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:49.479835 28584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:49.479835 28587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:49.480379 28579 server_base.cc:1034] running on GCE node
I20260927 04:03:49.480613 28579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:49.480888 28579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:49.482053 28579 hybrid_clock.cc:648] HybridClock initialized: now 1790481829482047 us; error 41 us; skew 500 ppm
Sep 27 04:03:49 dist-test-slave-zqrn krb5kdc[28338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481829, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:49.484716 28579 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:03:49.485222 28579 webserver.cc:460] Webserver started at http://127.17.184.194:33317/ using document root <none> and password file <none>
I20260927 04:03:49.485493 28579 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:49.485569 28579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:49.485836 28579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:49.487090 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "00fd10bd6ba043c0a7bc8a3a908789b1"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "0854ee25713a6e405339dd65f4a56682"
server_key_iv: "d41defae7e9fcbacb47915aaf9ad59b3"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.487478 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "00fd10bd6ba043c0a7bc8a3a908789b1"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "0854ee25713a6e405339dd65f4a56682"
server_key_iv: "d41defae7e9fcbacb47915aaf9ad59b3"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.488903 28579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:03:49.489749 28594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.489934 28579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:03:49.490046 28579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "00fd10bd6ba043c0a7bc8a3a908789b1"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "0854ee25713a6e405339dd65f4a56682"
server_key_iv: "d41defae7e9fcbacb47915aaf9ad59b3"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.490146 28579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:49.505122 28579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:49.507573 28579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:49.507768 28579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:49.508055 28579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:49.508419 28579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:49.508479 28579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.508523 28579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:49.508556 28579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.515208 28579 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33657
I20260927 04:03:49.515254 28707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33657 every 8 connection(s)
I20260927 04:03:49.515609 28579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:03:49 dist-test-slave-zqrn krb5kdc[28338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481829, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:49.522614 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 28579
I20260927 04:03:49.522711 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:03:49.522998 18147 external_mini_cluster.cc:1428] Setting key 227ec40f5b10446a7913f74fde8f4ca8
I20260927 04:03:49.526234 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.516444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:35753 (local address 127.17.184.254:37071)
0927 04:03:49.516526 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.516529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.516761 (+ 232us) server_negotiation.cc:407] Connection header received
0927 04:03:49.517262 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.517264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.517275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.517344 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:49.518345 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.519007 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.520012 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.520224 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.520285 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.522877 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:49.522878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:49.522878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:49.522886 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:49.524701 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.525128 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.525129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.525129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.525204 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.525540 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.525541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.525541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.525658 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:49.525714 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.526071 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.526153 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 04:03:49.526515 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.516412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37071 (local address 127.17.184.194:35753)
0927 04:03:49.516603 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.516642 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:49.517136 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:49.517418 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:49.517421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:49.517995 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:49.518213 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.518218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.519085 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.519086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.519922 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.519924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.520327 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.520328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.520448 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.521107 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:49.521116 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:49.522766 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:49.524803 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:49.524810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:49.524812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:49.525052 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:49.525304 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:49.525305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:49.525306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:49.525462 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:49.525729 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:49.525735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:49.525917 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 04:03:49.526042 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 04:03:49.526253 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
I20260927 04:03:49.527098 28708 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37071
I20260927 04:03:49.527207 28708 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:49.527403 28708 heartbeater.cc:511] Master 127.17.184.254:37071 requested a full tablet report, sending...
I20260927 04:03:49.527809 28383 ts_manager.cc:194] Registered new tserver with Master: 00fd10bd6ba043c0a7bc8a3a908789b1 (127.17.184.194:33657)
I20260927 04:03:49.528185 28383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:35753
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:03:49.554104 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37071
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:40559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:03:49.639869 28713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:03:49.640072 28713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:03:49.640095 28713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:03:49.641189 28713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:03:49.641225 28713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:03:49.641251 28713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:03:49.642627 28713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:40559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:03:49.642812 28713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:03:49.643019 28713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:03:49.648448 28713 server_base.cc:1034] running on GCE node
W20260927 04:03:49.648370 28718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:49.648370 28719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:03:49.648392 28721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:03:49.649008 28713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:03:49.649228 28713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:03:49.650384 28713 hybrid_clock.cc:648] HybridClock initialized: now 1790481829650369 us; error 30 us; skew 500 ppm
Sep 27 04:03:49 dist-test-slave-zqrn krb5kdc[28338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481829, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:03:49.653062 28713 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:03:49.653504 28713 webserver.cc:460] Webserver started at http://127.17.184.195:36573/ using document root <none> and password file <none>
I20260927 04:03:49.653753 28713 fs_manager.cc:359] Metadata directory not provided
I20260927 04:03:49.653822 28713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:03:49.653968 28713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:03:49.655138 28713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "86164aa6c0e84261835a74a6488b881d"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "cc9d08bb55e3da5508ca51c079224182"
server_key_iv: "400252d730ff6f0a33d42253366d46e8"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.655534 28713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "86164aa6c0e84261835a74a6488b881d"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "cc9d08bb55e3da5508ca51c079224182"
server_key_iv: "400252d730ff6f0a33d42253366d46e8"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.656890 28713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:49.657663 28728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.657857 28713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:03:49.657965 28713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "86164aa6c0e84261835a74a6488b881d"
format_stamp: "Formatted at 2026-09-27 04:03:49 on dist-test-slave-zqrn"
server_key: "cc9d08bb55e3da5508ca51c079224182"
server_key_iv: "400252d730ff6f0a33d42253366d46e8"
server_key_version: "encryptionkey@0"
I20260927 04:03:49.658039 28713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:03:49.704609 28713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:03:49.707078 28713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:03:49.707249 28713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:03:49.707504 28713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:03:49.707840 28713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:03:49.707895 28713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.707939 28713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:03:49.707970 28713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:03:49.714058 28713 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:40643
I20260927 04:03:49.714115 28841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:40643 every 8 connection(s)
I20260927 04:03:49.714439 28713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:03:49.720016 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 28713
I20260927 04:03:49.720136 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:03:49.720394 18147 external_mini_cluster.cc:1428] Setting key e6b722917fc9f07f22e07bea53086ba8
Sep 27 04:03:49 dist-test-slave-zqrn krb5kdc[28338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481829, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:03:49.724370 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.715259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:46225 (local address 127.17.184.254:37071)
0927 04:03:49.715318 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.715320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.715536 (+ 216us) server_negotiation.cc:407] Connection header received
0927 04:03:49.716129 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.716131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.716144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.716189 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:49.717198 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.717711 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.718738 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.718986 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.719071 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.721635 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:49.721636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:49.721637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:49.721643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:49.723041 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.723444 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.723445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.723445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.723500 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.723754 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.723754 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.723755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.723868 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:49.723922 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.724211 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.724299 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 04:03:49.724498 28844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.715145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37071 (local address 127.17.184.195:46225)
0927 04:03:49.715374 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.715424 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:49.716011 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:49.716269 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:49.716273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:49.716889 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:03:49.717086 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.717091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.717808 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.717809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.718633 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.718635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.719081 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.719082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.719227 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.719704 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:03:49.719712 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:03:49.721484 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:03:49.723115 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:49.723122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:49.723124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:49.723370 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:49.723564 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:03:49.723565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:03:49.723565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:03:49.723666 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:03:49.723925 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:03:49.723930 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:03:49.724053 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 04:03:49.724145 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 04:03:49.724274 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":121,"threads_started":1}
I20260927 04:03:49.725137 28842 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37071
I20260927 04:03:49.725268 28842 heartbeater.cc:464] Registering TS with master...
I20260927 04:03:49.725479 28842 heartbeater.cc:511] Master 127.17.184.254:37071 requested a full tablet report, sending...
I20260927 04:03:49.725888 28383 ts_manager.cc:194] Registered new tserver with Master: 86164aa6c0e84261835a74a6488b881d (127.17.184.195:40643)
I20260927 04:03:49.726255 28383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:46225
I20260927 04:03:49.732609 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:03:49.740301 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.733935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.17.184.254:37071)
0927 04:03:49.734008 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.734011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.734087 (+ 76us) server_negotiation.cc:407] Connection header received
0927 04:03:49.734178 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.734179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.734190 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.734246 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:03:49.734839 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.735455 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.736168 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.736364 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.736450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.737659 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:03:49.737661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:03:49.737661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:03:49.737668 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:03:49.739123 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.739488 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.739489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.739490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.739548 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:03:49.739742 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:03:49.739743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:03:49.739743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:03:49.739902 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:03:49.740009 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.740118 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.740196 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:03:49.741923 28383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:03:49.743634 28383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:03:49.750322 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.746103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.17.184.193:36019)
0927 04:03:49.746373 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.746379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.746511 (+ 132us) server_negotiation.cc:407] Connection header received
0927 04:03:49.746558 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.746560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.746577 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.746643 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:49.746850 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.747862 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.749142 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.749887 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.749969 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.750028 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.750176 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.750219 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260927 04:03:49.750320 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.746016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33657 (local address 127.0.0.1:46630)
0927 04:03:49.746312 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.746354 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:49.746423 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:49.746633 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:49.746636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:49.746639 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:49.746720 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.746722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.748110 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.748111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.749081 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.749083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.749988 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.749989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.750054 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.750072 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 04:03:49.750138 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 04:03:49.750200 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
I20260927 04:03:49.750322 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.746017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:36019 (local address 127.0.0.1:37410)
0927 04:03:49.746371 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.746380 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:49.746423 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:49.746722 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:49.746722 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:49.746724 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:49.746767 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.746768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.747962 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.747963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.749075 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.749078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.749973 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.749974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.750052 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.750065 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 04:03:49.750143 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 04:03:49.750204 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260927 04:03:49.750594 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.746228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:40643 (local address 127.0.0.1:52368)
0927 04:03:49.746522 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.746531 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:49.746566 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:49.747186 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:49.747187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:49.747189 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:49.747222 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.747223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.748453 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.748454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.749506 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.749508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.750344 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.750345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.750406 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.750409 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:03:49.750461 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 04:03:49.750534 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":100,"threads_started":1}
I20260927 04:03:49.750741 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.746107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.17.184.194:33657)
0927 04:03:49.746455 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.746458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.746468 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:49.746507 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.746509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.746526 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.746628 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:49.746809 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.747989 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.749184 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.749887 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.750546 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.750608 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.750632 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.750668 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":88,"threads_started":1}
I20260927 04:03:49.750895 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.746334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52368 (local address 127.17.184.195:40643)
0927 04:03:49.746970 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.746974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.746984 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:49.747022 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.747024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.747040 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.747110 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:49.747299 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.748350 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.749597 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.750188 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.750258 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.750311 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.750753 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.750797 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":86,"threads_started":1}
I20260927 04:03:49.751535 28508 tablet_service.cc:1467] Processing CreateTablet for tablet e158582290af4e2fa5359a09ca57a163 (DEFAULT_TABLE table=test-table [id=a8b50a6d6c1f4b3a9e2faa92d48e5a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:49.751533 28642 tablet_service.cc:1467] Processing CreateTablet for tablet e158582290af4e2fa5359a09ca57a163 (DEFAULT_TABLE table=test-table [id=a8b50a6d6c1f4b3a9e2faa92d48e5a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:49.751533 28776 tablet_service.cc:1467] Processing CreateTablet for tablet e158582290af4e2fa5359a09ca57a163 (DEFAULT_TABLE table=test-table [id=a8b50a6d6c1f4b3a9e2faa92d48e5a19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:03:49.751876 28508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e158582290af4e2fa5359a09ca57a163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:49.751878 28642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e158582290af4e2fa5359a09ca57a163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:49.751878 28776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e158582290af4e2fa5359a09ca57a163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:03:49.754374 28859 tablet_bootstrap.cc:492] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207: Bootstrap starting.
I20260927 04:03:49.755105 28859 tablet_bootstrap.cc:654] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:49.755584 28859 log.cc:824] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:49.755896 28861 tablet_bootstrap.cc:492] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Bootstrap starting.
I20260927 04:03:49.756385 28859 tablet_bootstrap.cc:492] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207: No bootstrap required, opened a new log
I20260927 04:03:49.756461 28859 ts_tablet_manager.cc:1397] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:03:49.756520 28861 tablet_bootstrap.cc:654] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:49.757114 28861 log.cc:824] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:49.757452 28860 tablet_bootstrap.cc:492] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Bootstrap starting.
I20260927 04:03:49.757817 28861 tablet_bootstrap.cc:492] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: No bootstrap required, opened a new log
I20260927 04:03:49.757897 28861 ts_tablet_manager.cc:1397] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:49.758059 28860 tablet_bootstrap.cc:654] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Neither blocks nor log segments found. Creating new log.
I20260927 04:03:49.758592 28860 log.cc:824] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:03:49.758694 28859 raft_consensus.cc:348] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.758880 28859 raft_consensus.cc:374] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:49.758962 28859 raft_consensus.cc:729] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 204ddd710d444694aa88f5acff0c0207, State: Initialized, Role: FOLLOWER
I20260927 04:03:49.759076 28859 consensus_queue.cc:260] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.759382 28859 ts_tablet_manager.cc:1428] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260927 04:03:49.759393 28574 heartbeater.cc:503] Master 127.17.184.254:37071 was elected leader, sending a full tablet report...
I20260927 04:03:49.759646 28860 tablet_bootstrap.cc:492] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: No bootstrap required, opened a new log
I20260927 04:03:49.759752 28860 ts_tablet_manager.cc:1397] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:49.759850 28861 raft_consensus.cc:348] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.760007 28861 raft_consensus.cc:374] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:49.760039 28861 raft_consensus.cc:729] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00fd10bd6ba043c0a7bc8a3a908789b1, State: Initialized, Role: FOLLOWER
I20260927 04:03:49.760150 28861 consensus_queue.cc:260] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.760412 28861 ts_tablet_manager.cc:1428] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 04:03:49.760464 28708 heartbeater.cc:503] Master 127.17.184.254:37071 was elected leader, sending a full tablet report...
I20260927 04:03:49.761797 28860 raft_consensus.cc:348] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.761917 28860 raft_consensus.cc:374] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:03:49.761937 28860 raft_consensus.cc:729] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86164aa6c0e84261835a74a6488b881d, State: Initialized, Role: FOLLOWER
I20260927 04:03:49.762061 28860 consensus_queue.cc:260] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.762326 28860 ts_tablet_manager.cc:1428] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:03:49.762496 28842 heartbeater.cc:503] Master 127.17.184.254:37071 was elected leader, sending a full tablet report...
W20260927 04:03:49.766179 28709 tablet.cc:2369] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:03:49.856959 28575 tablet.cc:2369] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:03:49.965081 28843 tablet.cc:2369] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:03:49.966107 28866 raft_consensus.cc:482] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:03:49.966219 28866 raft_consensus.cc:504] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.966657 28866 leader_election.cc:288] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 204ddd710d444694aa88f5acff0c0207 (127.17.184.193:36019), 86164aa6c0e84261835a74a6488b881d (127.17.184.195:40643)
I20260927 04:03:49.969709 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.966986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:46165 (local address 127.17.184.193:36019)
0927 04:03:49.967046 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.967049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.967060 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:03:49.967099 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.967100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.967116 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.967213 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:49.967420 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.967920 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.968945 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.969322 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.969419 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.969429 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.969593 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.969635 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 04:03:49.969758 28710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.966844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:36019 (local address 127.17.184.194:46165)
0927 04:03:49.966926 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.966939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:49.966986 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:49.967218 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:49.967219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:49.967221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:49.967327 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.967329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.968009 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.968010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.968848 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.968851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.969416 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.969417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.969513 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.969517 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:03:49.969585 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 04:03:49.969626 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":40}
I20260927 04:03:49.969759 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.967074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:49615 (local address 127.17.184.195:40643)
0927 04:03:49.967165 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.967167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.967258 (+ 91us) server_negotiation.cc:407] Connection header received
0927 04:03:49.967326 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.967327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.967336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.967422 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:03:49.967573 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.968042 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.968996 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.969382 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.969477 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.969485 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.969649 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.969688 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:03:49.969774 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.966986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:40643 (local address 127.17.184.194:49615)
0927 04:03:49.967187 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.967196 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:03:49.967271 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:03:49.967430 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:03:49.967431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:03:49.967432 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:03:49.967499 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.967502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.968142 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.968143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.968895 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:03:49.968897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.969478 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.969479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:03:49.969551 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.969561 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 04:03:49.969641 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 04:03:49.969681 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":127,"threads_started":1}
I20260927 04:03:49.970175 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e158582290af4e2fa5359a09ca57a163" candidate_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86164aa6c0e84261835a74a6488b881d" is_pre_election: true
I20260927 04:03:49.970244 28528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e158582290af4e2fa5359a09ca57a163" candidate_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204ddd710d444694aa88f5acff0c0207" is_pre_election: true
I20260927 04:03:49.970366 28528 raft_consensus.cc:2454] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fd10bd6ba043c0a7bc8a3a908789b1 in term 0.
I20260927 04:03:49.970366 28796 raft_consensus.cc:2454] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00fd10bd6ba043c0a7bc8a3a908789b1 in term 0.
I20260927 04:03:49.970584 28596 leader_election.cc:302] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fd10bd6ba043c0a7bc8a3a908789b1, 86164aa6c0e84261835a74a6488b881d; no voters:
I20260927 04:03:49.970767 28866 raft_consensus.cc:2781] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:03:49.970832 28866 raft_consensus.cc:482] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:03:49.970849 28866 raft_consensus.cc:3037] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:49.971617 28866 raft_consensus.cc:504] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.971757 28866 leader_election.cc:288] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [CANDIDATE]: Term 1 election: Requested vote from peers 204ddd710d444694aa88f5acff0c0207 (127.17.184.193:36019), 86164aa6c0e84261835a74a6488b881d (127.17.184.195:40643)
I20260927 04:03:49.971925 28528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e158582290af4e2fa5359a09ca57a163" candidate_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "204ddd710d444694aa88f5acff0c0207"
I20260927 04:03:49.971963 28796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e158582290af4e2fa5359a09ca57a163" candidate_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86164aa6c0e84261835a74a6488b881d"
I20260927 04:03:49.972008 28528 raft_consensus.cc:3037] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:49.972082 28796 raft_consensus.cc:3037] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:03:49.972743 28528 raft_consensus.cc:2454] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fd10bd6ba043c0a7bc8a3a908789b1 in term 1.
I20260927 04:03:49.972900 28596 leader_election.cc:302] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00fd10bd6ba043c0a7bc8a3a908789b1, 204ddd710d444694aa88f5acff0c0207; no voters:
I20260927 04:03:49.972913 28796 raft_consensus.cc:2454] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00fd10bd6ba043c0a7bc8a3a908789b1 in term 1.
I20260927 04:03:49.973003 28866 raft_consensus.cc:2781] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:03:49.973192 28866 raft_consensus.cc:686] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 1 LEADER]: Becoming Leader. State: Replica: 00fd10bd6ba043c0a7bc8a3a908789b1, State: Running, Role: LEADER
I20260927 04:03:49.973302 28866 consensus_queue.cc:237] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } }
I20260927 04:03:49.974208 28383 catalog_manager.cc:5495] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00fd10bd6ba043c0a7bc8a3a908789b1 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00fd10bd6ba043c0a7bc8a3a908789b1" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 } health_report { overall_health: UNKNOWN } } }
I20260927 04:03:49.982990 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.980321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.17.184.194:33657)
0927 04:03:49.980409 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0927 04:03:49.980412 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:03:49.980422 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:03:49.980507 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:03:49.980508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:03:49.980534 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:03:49.980623 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:03:49.980846 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.981342 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.982126 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:03:49.982334 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:03:49.982421 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:03:49.982600 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:03:49.982712 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:03:49.982817 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 04:03:49.982836 (+ 19us) server_negotiation.cc:299] Negotiation successful
0927 04:03:49.982855 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 04:03:49.984931 28796 raft_consensus.cc:1261] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 1 FOLLOWER]: Refusing update from remote peer 00fd10bd6ba043c0a7bc8a3a908789b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:49.984951 28528 raft_consensus.cc:1261] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [term 1 FOLLOWER]: Refusing update from remote peer 00fd10bd6ba043c0a7bc8a3a908789b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:03:49.985285 28869 consensus_queue.cc:1035] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86164aa6c0e84261835a74a6488b881d" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 40643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:49.985430 28866 consensus_queue.cc:1035] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "204ddd710d444694aa88f5acff0c0207" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:03:49.987871 28872 mvcc.cc:205] Tried to move back new op lower bound from 7333813575615176704 to 7333813575571492864. Current Snapshot: MvccSnapshot[applied={T|T < 7333813575615176704}]
I20260927 04:03:49.989033 28874 mvcc.cc:205] Tried to move back new op lower bound from 7333813575615176704 to 7333813575571492864. Current Snapshot: MvccSnapshot[applied={T|T < 7333813575615176704 or (T in {7333813575615176704})}]
I20260927 04:03:49.990272 28383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:49.990372 28383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:03:49.991212 28383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7afce2004d343be8efcc65635b6d79a: Sending DeleteTablet for 3 replicas of tablet e158582290af4e2fa5359a09ca57a163
I20260927 04:03:49.991518 28642 tablet_service.cc:1514] Processing DeleteTablet for tablet e158582290af4e2fa5359a09ca57a163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:49 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:46630
I20260927 04:03:49.991654 28508 tablet_service.cc:1514] Processing DeleteTablet for tablet e158582290af4e2fa5359a09ca57a163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:49 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:37410
I20260927 04:03:49.991753 28776 tablet_service.cc:1514] Processing DeleteTablet for tablet e158582290af4e2fa5359a09ca57a163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:03:49 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:52368
I20260927 04:03:49.991899 28882 tablet_replica.cc:330] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207: stopping tablet replica
I20260927 04:03:49.992000 28882 raft_consensus.cc:2229] T e158582290af4e2fa5359a09ca57a163 P 204ddd710d444694aa88f5acff0c0207 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:49.992003 28883 tablet_replica.cc:330] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: stopping tablet replica
I20260927 04:03:49.992101 28883 raft_consensus.cc:2229] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:03:49.992206 28883 raft_consensus.cc:2258] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:49.992317 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 28445
I20260927 04:03:49.992410 28881 tablet_replica.cc:330] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: stopping tablet replica
I20260927 04:03:49.992530 28881 raft_consensus.cc:2229] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:03:49.992728 28883 ts_tablet_manager.cc:1905] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:49.993763 28881 raft_consensus.cc:2258] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:03:49.994263 28881 ts_tablet_manager.cc:1905] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:03:49.995298 28883 ts_tablet_manager.cc:1918] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:49.995379 28883 log.cc:1196] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/e158582290af4e2fa5359a09ca57a163
I20260927 04:03:49.995692 28883 ts_tablet_manager.cc:1939] T e158582290af4e2fa5359a09ca57a163 P 86164aa6c0e84261835a74a6488b881d: Deleting consensus metadata
I20260927 04:03:49.996130 28368 catalog_manager.cc:4841] TS 86164aa6c0e84261835a74a6488b881d (127.17.184.195:40643): tablet e158582290af4e2fa5359a09ca57a163 (table test-table [id=a8b50a6d6c1f4b3a9e2faa92d48e5a19]) successfully deleted
I20260927 04:03:49.996567 28881 ts_tablet_manager.cc:1918] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:03:49.996654 28881 log.cc:1196] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/e158582290af4e2fa5359a09ca57a163
I20260927 04:03:49.996899 28881 ts_tablet_manager.cc:1939] T e158582290af4e2fa5359a09ca57a163 P 00fd10bd6ba043c0a7bc8a3a908789b1: Deleting consensus metadata
I20260927 04:03:49.997830 28369 catalog_manager.cc:4841] TS 00fd10bd6ba043c0a7bc8a3a908789b1 (127.17.184.194:33657): tablet e158582290af4e2fa5359a09ca57a163 (table test-table [id=a8b50a6d6c1f4b3a9e2faa92d48e5a19]) successfully deleted
W20260927 04:03:49.999328 28368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:36019 (error 108)
I20260927 04:03:50.000018 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:03:49.999770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:36019 (local address 127.0.0.1:37418)
0927 04:03:49.999833 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 04:03:49.999885 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:36019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:03:50.000008 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 28579
W20260927 04:03:50.000427 28368 catalog_manager.cc:4584] TS 204ddd710d444694aa88f5acff0c0207 (127.17.184.193:36019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e158582290af4e2fa5359a09ca57a163: Network error: Client connection negotiation failed: client connection to 127.17.184.193:36019: connect: Connection refused (error 111)
I20260927 04:03:50.006011 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 28713
I20260927 04:03:50.012377 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 28351
2026-09-27T04:03:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2199 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:03:50 dist-test-slave-zqrn krb5kdc[28887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:03:50 dist-test-slave-zqrn krb5kdc[28887](info): set up 2 sockets
Sep 27 04:03:50 dist-test-slave-zqrn krb5kdc[28887](info): commencing operation
krb5kdc: starting...
W20260927 04:03:51.144009 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:03:51 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:03:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:03:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.184.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.184.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/rangeradmin_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/rangeradmin_127.17.184.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.184.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.184.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/rangerlookup_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/rangerlookup_127.17.184.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.184.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.184.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/HTTP_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/HTTP_127.17.184.212@KRBTEST.COM.keytab.
I20260927 04:03:51.335085 18147 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJqOnmp/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/postgres -l logfile start
2026-09-27 04:03:54.338 UTC [28917] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 04:03:54.338 UTC [28917] LOG: listening on IPv4 address "127.17.184.212", port 49649
2026-09-27 04:03:54.342 UTC [28917] LOG: listening on Unix socket "/tmp/.s.PGSQL.49649"
2026-09-27 04:03:54.365 UTC [28920] LOG: database system was shut down at 2026-09-27 04:03:52 UTC
2026-09-27 04:03:54.369 UTC [28917] LOG: database system is ready to accept connections
I20260927 04:03:55.335876 18147 mini_postgres.cc:96] Postgres bound to 49649
2026-09-27 04:03:55.341 UTC [28928] FATAL: database "slave" does not exist
127.17.184.212:49649 - accepting connections
I20260927 04:03:55.342480 18147 mini_ranger.cc:161] Starting Ranger...
I20260927 04:03:55.357391 18147 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 04:03:55.909210 18147 mini_ranger.cc:87] Created ranger Postgres database
I20260927 04:03:55.909296 18147 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-admin
2026-09-27 04:03:56,191 [I] DB FLAVOR :POSTGRES
2026-09-27 04:03:56,191 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 04:03:56,192 [I] Checking connection..
2026-09-27 04:03:56,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 04:03:56,459 [I] Checking connection passed.
2026-09-27 04:03:56,459 [I] --------- Verifying version history table ---------
2026-09-27 04:03:56,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:03:56,722 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 04:03:56,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:03:56,972 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 04:03:56,972 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 04:03:56,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 04:03:57.217 UTC [28992] WARNING: there is no transaction in progress
2026-09-27 04:03:57,224 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 04:03:57,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:03:57,476 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 04:03:57,477 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 04:03:57,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 04:03:57,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 04:03:57,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zqrn.c.gcp-upstream.internal','N') ;"
2026-09-27 04:03:58,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 04:03:58,494 [I] Table x_portal_user does not exist in database ranger
2026-09-27 04:03:58,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 04:03:58,744 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 04:03:58,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 04:03:58,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 04:03:59,244 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 04:03:59,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 04:04:00.435 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.457 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.477 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.496 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.514 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.535 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.599 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.608 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.625 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.637 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.651 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.668 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.679 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.687 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.695 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:00.703 UTC [29100] WARNING: there is no transaction in progress
2026-09-27 04:04:01,531 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 04:04:01,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 04:04:01,786 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 04:04:01,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 04:04:02,024 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 04:04:02,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 04:04:02,305 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 04:04:02,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 04:04:02,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 04:04:02,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zqrn.c.gcp-upstream.internal';"
2026-09-27 04:04:03,046 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 04:04:03,046 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 04:04:03,284 [I] DB_PATCHES have already been applied
I20260927 04:04:03.291038 18147 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJqOnmp/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 04:04:03.291114 18147 mini_ranger.cc:193] Using host: 127.17.184.212
I20260927 04:04:03.294445 18147 mini_ranger.cc:227] Ranger admin URL: http://127.17.184.212:44541
Sep 27, 2026 4:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:04:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:04:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 04:04:04.415104 18147 mini_ranger.cc:160] Time spent starting Ranger: real 9.073s user 0.000s sys 0.009s
Sep 27, 2026 4:04:24 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:04:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 4:04:24 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:04:24 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:04:25 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 04:04:26.244089 18147 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.184.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.184.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/rangerkms_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/rangerkms_127.17.184.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.184.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.184.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.184.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/HTTP_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.184.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/HTTP_127.17.184.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 04:04:26 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 04:04:26.355173 18147 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 04:04:26.366242 18147 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 04:04:27.430538 18147 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 04:04:27.430624 18147 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms
I20260927 04:04:27.430639 18147 mini_ranger_kms.cc:227] Using postgres at 127.17.184.212:49649
2026-09-27 04:04:27,613 [I] DB FLAVOR :POSTGRES
2026-09-27 04:04:27,613 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 04:04:27,613 [I] Checking connection
2026-09-27 04:04:27,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 04:04:27,859 [I] connection success
2026-09-27 04:04:27,859 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 04:04:27,859 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 04:04:27,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 04:04:28,126 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 04:04:28,126 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 04:04:28,127 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 04:04:28,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:49649/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 04:04:28,408 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 04:04:28.901346 18147 mini_ranger_kms.cc:310] Created kms service
I20260927 04:04:29.117487 18147 mini_ranger_kms.cc:326] Created kudu user
I20260927 04:04:29.169433 18147 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 04:04:29 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481866, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:04:29.560675 18147 mini_ranger_kms.cc:384] Added ranger policy
I20260927 04:04:29.560801 18147 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 04:04:29.560824 18147 mini_ranger_kms.cc:240] Using host: 127.17.184.212
I20260927 04:04:29.564221 18147 mini_ranger_kms.cc:276] Ranger KMS PID: 29280
I20260927 04:04:29.564369 18147 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.184.212:44383
Sep 27, 2026 4:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:04:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:04:31 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 04:04:31.963899 18147 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.609s user 0.015s sys 0.150s
I20260927 04:04:31.964012 18147 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 04:04:31.964099 18147 mini_ranger_kms.cc:401] 127.17.184.212:44383/kms/v1/keys
Sep 27 04:04:32 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.184.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 04:04:32 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.184.212@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
Sep 27 04:04:32 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481872, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.184.212@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
Sep 27 04:04:35 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:04:35.283175 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:37451
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:37451
--ranger_config_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:44383/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:04:35.387660 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:04:35.387881 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:04:35.387904 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:04:35.389140 29331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:04:35.389190 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:04:35.389210 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:04:35.389225 29331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:04:35.389236 29331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:04:35.390851 29331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.184.212:44383/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:37451
--ranger_config_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:37451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:04:35.391142 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:04:35.391410 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:04:35.396984 29331 server_base.cc:1034] running on GCE node
W20260927 04:04:35.397013 29339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:04:35.397056 29337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:04:35.397024 29336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:04:35.397435 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:04:35.397795 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:04:35.398960 29331 hybrid_clock.cc:648] HybridClock initialized: now 1790481875398950 us; error 39 us; skew 500 ppm
Sep 27 04:04:35 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481875, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:04:35.402108 29331 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:04:35.402643 29331 webserver.cc:460] Webserver started at http://127.17.184.254:35451/ using document root <none> and password file <none>
I20260927 04:04:35.402936 29331 fs_manager.cc:359] Metadata directory not provided
I20260927 04:04:35.402993 29331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:04:35.403146 29331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:04:35 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481875, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:04:35.451809 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "6674dcaf3ad1471a859ce8a88f9a15b6"
format_stamp: "Formatted at 2026-09-27 04:04:35 on dist-test-slave-zqrn"
server_key: "585d35e592ba5347680fa1001e24e3d8"
server_key_iv: "bf87719b7e013f26a25f6becfad368f4"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:35.452245 29331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "6674dcaf3ad1471a859ce8a88f9a15b6"
format_stamp: "Formatted at 2026-09-27 04:04:35 on dist-test-slave-zqrn"
server_key: "585d35e592ba5347680fa1001e24e3d8"
server_key_iv: "bf87719b7e013f26a25f6becfad368f4"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:35.453756 29331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:04:35.465296 29346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:35.465517 29331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:04:35.465610 29331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "6674dcaf3ad1471a859ce8a88f9a15b6"
format_stamp: "Formatted at 2026-09-27 04:04:35 on dist-test-slave-zqrn"
server_key: "585d35e592ba5347680fa1001e24e3d8"
server_key_iv: "bf87719b7e013f26a25f6becfad368f4"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:35.465739 29331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:04:35.482776 29331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:04:35.484565 29331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:04:35.484740 29331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:04:35.489753 29331 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:37451
I20260927 04:04:35.489815 29398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:37451 every 8 connection(s)
I20260927 04:04:35.490195 29331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:04:35.490492 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 29331
I20260927 04:04:35.490628 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:04:35.490841 18147 external_mini_cluster.cc:1428] Setting key 72771fcfb890796d42258b2a340ec9f2
I20260927 04:04:35.491009 29399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:04:35.494431 29399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6: Bootstrap starting.
I20260927 04:04:35.495163 29399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6: Neither blocks nor log segments found. Creating new log.
I20260927 04:04:35.495494 29399 log.cc:824] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6: Log is configured to *not* fsync() on all Append() calls
I20260927 04:04:35.496246 29399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6: No bootstrap required, opened a new log
Sep 27 04:04:35 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:04:35.498634 29399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6674dcaf3ad1471a859ce8a88f9a15b6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37451 } }
I20260927 04:04:35.498819 29399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:04:35.498867 29399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6674dcaf3ad1471a859ce8a88f9a15b6, State: Initialized, Role: FOLLOWER
I20260927 04:04:35.499030 29399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6674dcaf3ad1471a859ce8a88f9a15b6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37451 } }
I20260927 04:04:35.499133 29399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:04:35.499177 29399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:04:35.499230 29399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:04:35.499986 29399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6674dcaf3ad1471a859ce8a88f9a15b6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37451 } }
I20260927 04:04:35.500105 29399 leader_election.cc:302] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6674dcaf3ad1471a859ce8a88f9a15b6; no voters:
I20260927 04:04:35.500319 29399 leader_election.cc:288] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:04:35.500442 29404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:04:35.500638 29399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:04:35.500665 29404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [term 1 LEADER]: Becoming Leader. State: Replica: 6674dcaf3ad1471a859ce8a88f9a15b6, State: Running, Role: LEADER
I20260927 04:04:35.500922 29404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6674dcaf3ad1471a859ce8a88f9a15b6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37451 } }
I20260927 04:04:35.502095 29405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6674dcaf3ad1471a859ce8a88f9a15b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6674dcaf3ad1471a859ce8a88f9a15b6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37451 } } }
I20260927 04:04:35.502228 29405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [sys.catalog]: This master's current role is: LEADER
I20260927 04:04:35.502709 29399 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 04:04:35.503088 29406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6674dcaf3ad1471a859ce8a88f9a15b6. Latest consensus state: current_term: 1 leader_uuid: "6674dcaf3ad1471a859ce8a88f9a15b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6674dcaf3ad1471a859ce8a88f9a15b6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 37451 } } }
I20260927 04:04:35.503147 29419 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:04:35.503165 29406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6 [sys.catalog]: This master's current role is: LEADER
I20260927 04:04:35.505877 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:35.491411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.17.184.254:37451)
0927 04:04:35.491620 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:35.491631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:35.491667 (+ 36us) server_negotiation.cc:407] Connection header received
0927 04:04:35.492295 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:35.492304 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:35.492371 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:35.492549 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:04:35.493434 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:35.494349 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:35.495454 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:35.495935 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:35.496019 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:35.499271 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:04:35.499274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:04:35.499275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:04:35.499287 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:04:35.501264 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:35.501782 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:35.501786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:35.501786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:35.501885 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:35.502124 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:35.502125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:35.502126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:35.502437 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:04:35.503147 (+ 710us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:35.503228 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 04:04:35.505189 (+ 1961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260927 04:04:35.506069 29419 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:04:35.509687 29419 catalog_manager.cc:1347] Generated new cluster ID: f04d3c80a88e4d2b818cb7ea115eddfa
I20260927 04:04:35.509759 29419 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:04:35.525717 29419 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:04:35.526508 29419 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:04:35.534458 29419 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6: Generated new TSK 0
I20260927 04:04:35.535303 29419 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:04:36.319155 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:44383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37451
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:04:36.434306 29444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:04:36.434588 29444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:04:36.434618 29444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:04:36.436028 29444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:04:36.436080 29444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:04:36.436111 29444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:04:36.437963 29444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.184.212:44383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:04:36.438282 29444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:04:36.438565 29444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:04:36.444803 29452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:04:36.444949 29450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:04:36.444803 29449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:04:36.445549 29444 server_base.cc:1034] running on GCE node
I20260927 04:04:36.445801 29444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:04:36.446125 29444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:04:36.447633 29444 hybrid_clock.cc:648] HybridClock initialized: now 1790481876447612 us; error 30 us; skew 500 ppm
Sep 27 04:04:36 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481876, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:04:36.451814 29444 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:04:36.452561 29444 webserver.cc:460] Webserver started at http://127.17.184.193:44031/ using document root <none> and password file <none>
I20260927 04:04:36.452867 29444 fs_manager.cc:359] Metadata directory not provided
I20260927 04:04:36.452947 29444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:04:36.453114 29444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:04:36 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481876, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:04:36.474094 29444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "c0a599449ea14693a6b80486155a6012"
format_stamp: "Formatted at 2026-09-27 04:04:36 on dist-test-slave-zqrn"
server_key: "0dcd71a8f75c3f3455db412eb2b5fd6f"
server_key_iv: "581b0e9ea0a072123a40a4b7c4daa7d2"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:36.474690 29444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "c0a599449ea14693a6b80486155a6012"
format_stamp: "Formatted at 2026-09-27 04:04:36 on dist-test-slave-zqrn"
server_key: "0dcd71a8f75c3f3455db412eb2b5fd6f"
server_key_iv: "581b0e9ea0a072123a40a4b7c4daa7d2"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:36.478201 29444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 04:04:36.499817 29461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:36.500058 29444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:04:36.500195 29444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "c0a599449ea14693a6b80486155a6012"
format_stamp: "Formatted at 2026-09-27 04:04:36 on dist-test-slave-zqrn"
server_key: "0dcd71a8f75c3f3455db412eb2b5fd6f"
server_key_iv: "581b0e9ea0a072123a40a4b7c4daa7d2"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:36.500317 29444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:04:36.532071 29444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:04:36.534981 29444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:04:36.535218 29444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:04:36.535518 29444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:04:36.536329 29444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:04:36.536434 29444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:36.536542 29444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:04:36.536625 29444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:36.545382 29444 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:34909
I20260927 04:04:36.545444 29574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:34909 every 8 connection(s)
I20260927 04:04:36.545986 29444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:04:36.546222 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 29444
I20260927 04:04:36.546326 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:04:36.546500 18147 external_mini_cluster.cc:1428] Setting key 27e75b82dd76151e7ff16b04989fd745
Sep 27 04:04:36 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481876, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:04:36.565524 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:36.547286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37451 (local address 127.17.184.193:35247)
0927 04:04:36.549312 (+ 2026us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:36.549364 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 04:04:36.550047 (+ 683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:04:36.551728 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:04:36.551734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:04:36.552051 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:04:36.552386 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:36.552393 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:36.553454 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:36.553456 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:36.554783 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:36.554786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:36.555345 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:36.555346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:36.555505 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:36.559082 (+ 3577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:04:36.559096 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:04:36.561058 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:04:36.563534 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:04:36.563547 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:04:36.563551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:04:36.563926 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:04:36.564198 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:04:36.564199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:04:36.564199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:04:36.564358 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:04:36.564727 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:04:36.564735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:04:36.564929 (+ 194us) client_negotiation.cc:769] Sending connection context
0927 04:04:36.565069 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 04:04:36.565255 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1976,"thread_start_us":89,"threads_started":1}
I20260927 04:04:36.566473 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:36.547620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:35247 (local address 127.17.184.254:37451)
0927 04:04:36.551463 (+ 3843us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:36.551466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:36.551477 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:04:36.551505 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:36.551506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:36.551521 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:36.551602 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:04:36.552527 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:36.553330 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:36.554920 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:36.555232 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:36.555309 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:36.561228 (+ 5919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:04:36.561231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:04:36.561231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:04:36.561241 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:04:36.563359 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:36.564045 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:36.564046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:36.564046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:36.564121 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:36.564448 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:36.564449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:36.564450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:36.564621 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:04:36.564687 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:36.566249 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0927 04:04:36.566354 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3816,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
I20260927 04:04:36.568826 29575 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37451
I20260927 04:04:36.568959 29575 heartbeater.cc:464] Registering TS with master...
I20260927 04:04:36.569204 29575 heartbeater.cc:511] Master 127.17.184.254:37451 requested a full tablet report, sending...
I20260927 04:04:36.569927 29363 ts_manager.cc:194] Registered new tserver with Master: c0a599449ea14693a6b80486155a6012 (127.17.184.193:34909)
I20260927 04:04:36.570829 29363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:35247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:04:36.602267 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:44383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37451
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:04:36 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481876, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 04:04:36.752701 29581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:04:36.753118 29581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:04:36.753211 29581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:04:36.755527 29581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:04:36.755703 29581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:04:36.756214 29581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:04:36.759954 29581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.184.212:44383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:04:36.760563 29581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:04:36.760870 29581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:04:36.768730 29591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:04:36.768643 29588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:04:36.768643 29589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:04:36.769449 29581 server_base.cc:1034] running on GCE node
I20260927 04:04:36.769791 29581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:04:36.770514 29581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:04:36.771744 29581 hybrid_clock.cc:648] HybridClock initialized: now 1790481876771722 us; error 40 us; skew 500 ppm
Sep 27 04:04:36 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481876, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:04:36.775504 29581 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:04:36.776120 29581 webserver.cc:460] Webserver started at http://127.17.184.194:44397/ using document root <none> and password file <none>
I20260927 04:04:36.776409 29581 fs_manager.cc:359] Metadata directory not provided
I20260927 04:04:36.776471 29581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:04:36.776614 29581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:04:36 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481876, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:04:36.794150 29581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "171584f6ef73488cac6ef3f0cd824ea3"
format_stamp: "Formatted at 2026-09-27 04:04:36 on dist-test-slave-zqrn"
server_key: "8af5e7018cd2804a65eaac8b8aef885a"
server_key_iv: "645f4c452216203acca6a575bc654269"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:36.794785 29581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "171584f6ef73488cac6ef3f0cd824ea3"
format_stamp: "Formatted at 2026-09-27 04:04:36 on dist-test-slave-zqrn"
server_key: "8af5e7018cd2804a65eaac8b8aef885a"
server_key_iv: "645f4c452216203acca6a575bc654269"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:36.799672 29581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260927 04:04:36.821755 29599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:36.822038 29581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 04:04:36.822144 29581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "171584f6ef73488cac6ef3f0cd824ea3"
format_stamp: "Formatted at 2026-09-27 04:04:36 on dist-test-slave-zqrn"
server_key: "8af5e7018cd2804a65eaac8b8aef885a"
server_key_iv: "645f4c452216203acca6a575bc654269"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:36.822261 29581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:04:36.838950 29581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:04:36.842605 29581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:04:36.842839 29581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:04:36.843178 29581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:04:36.843571 29581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:04:36.843645 29581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:36.843708 29581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:04:36.843739 29581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:36.852026 29581 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:39557
I20260927 04:04:36.852126 29712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:39557 every 8 connection(s)
I20260927 04:04:36.852510 29581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:04:36.856993 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 29581
I20260927 04:04:36.857204 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:04:36.857456 18147 external_mini_cluster.cc:1428] Setting key a0dfcd2ba6f8aa604fc086a1a0c5a270
Sep 27 04:04:36 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481876, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:04:36.865624 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:36.853399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:50105 (local address 127.17.184.254:37451)
0927 04:04:36.853478 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:36.853481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:36.853710 (+ 229us) server_negotiation.cc:407] Connection header received
0927 04:04:36.854578 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:36.854579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:36.854598 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:36.854685 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:04:36.855549 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:36.856318 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:36.858085 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:36.858510 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:36.858593 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:36.861572 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:04:36.861575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:04:36.861576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:04:36.861585 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:04:36.863544 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:36.864259 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:36.864261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:36.864262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:36.864331 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:36.864671 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:36.864672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:36.864672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:36.864845 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:04:36.864896 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:36.865362 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 04:04:36.865475 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:04:36.866221 29715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:36.853322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37451 (local address 127.17.184.194:50105)
0927 04:04:36.853512 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:36.853556 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:04:36.854430 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:04:36.854848 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:04:36.854852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:04:36.855184 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:04:36.855414 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:36.855420 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:36.856448 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:36.856449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:36.857971 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:36.857974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:36.858611 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:36.858612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:36.858764 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:36.859485 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:04:36.859496 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:04:36.861453 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:04:36.863681 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:04:36.863693 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:04:36.863697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:04:36.864136 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:04:36.864421 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:04:36.864422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:04:36.864422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:04:36.864586 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:04:36.864951 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:04:36.864958 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:04:36.865179 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 04:04:36.865737 (+ 558us) client_negotiation.cc:240] Negotiation successful
0927 04:04:36.865934 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260927 04:04:36.866842 29713 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37451
I20260927 04:04:36.867012 29713 heartbeater.cc:464] Registering TS with master...
I20260927 04:04:36.867283 29713 heartbeater.cc:511] Master 127.17.184.254:37451 requested a full tablet report, sending...
I20260927 04:04:36.867856 29363 ts_manager.cc:194] Registered new tserver with Master: 171584f6ef73488cac6ef3f0cd824ea3 (127.17.184.194:39557)
I20260927 04:04:36.868477 29363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:50105
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:04:36.896291 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:44383/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37451
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:04:37.011636 29718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:04:37.012387 29718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:04:37.012439 29718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:04:37.015512 29718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:04:37.015682 29718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:04:37.015786 29718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:04:37.019351 29718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.184.212:44383/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:37451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:04:37.019964 29718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:04:37.020339 29718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:04:37.027928 29724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:04:37.028919 29723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:04:37.028950 29726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:04:37.030061 29718 server_base.cc:1034] running on GCE node
I20260927 04:04:37.030445 29718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:04:37.030994 29718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:04:37.032281 29718 hybrid_clock.cc:648] HybridClock initialized: now 1790481877032237 us; error 50 us; skew 500 ppm
Sep 27 04:04:37 dist-test-slave-zqrn krb5kdc[28887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481877, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:04:37.036904 29718 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:04:37.037686 29718 webserver.cc:460] Webserver started at http://127.17.184.195:38783/ using document root <none> and password file <none>
I20260927 04:04:37.038105 29718 fs_manager.cc:359] Metadata directory not provided
I20260927 04:04:37.038311 29718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:04:37.038625 29718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:04:37 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481877, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:04:37.055517 29718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "04372450365d47398e7c7f4f56ba2f18"
format_stamp: "Formatted at 2026-09-27 04:04:37 on dist-test-slave-zqrn"
server_key: "3dbceb67575faac3f11f1f75f0cfde98"
server_key_iv: "122d32df84ecf79799bad65e9b66ac00"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:37.056663 29718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "04372450365d47398e7c7f4f56ba2f18"
format_stamp: "Formatted at 2026-09-27 04:04:37 on dist-test-slave-zqrn"
server_key: "3dbceb67575faac3f11f1f75f0cfde98"
server_key_iv: "122d32df84ecf79799bad65e9b66ac00"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:37.059360 29718 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260927 04:04:37.089793 29733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:37.090054 29718 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 04:04:37.090360 29718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "04372450365d47398e7c7f4f56ba2f18"
format_stamp: "Formatted at 2026-09-27 04:04:37 on dist-test-slave-zqrn"
server_key: "3dbceb67575faac3f11f1f75f0cfde98"
server_key_iv: "122d32df84ecf79799bad65e9b66ac00"
server_key_version: "kuduclusterkey@0"
I20260927 04:04:37.090520 29718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:04:37.105777 29718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:04:37.108783 29718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:04:37.108966 29718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:04:37.109257 29718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:04:37.109639 29718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:04:37.109751 29718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:37.109830 29718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:04:37.109889 29718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:04:37.119634 29718 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:34325
I20260927 04:04:37.120227 29718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:04:37.121234 29846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:34325 every 8 connection(s)
I20260927 04:04:37.123843 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 29718
I20260927 04:04:37.123946 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:04:37.124130 18147 external_mini_cluster.cc:1428] Setting key 1796c14d7d7580e9db35355fdae5f4b2
Sep 27 04:04:37 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481877, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:04:37.136662 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.122222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:59117 (local address 127.17.184.254:37451)
0927 04:04:37.122271 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:37.122273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:37.122531 (+ 258us) server_negotiation.cc:407] Connection header received
0927 04:04:37.123309 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:37.123310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:37.123324 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:37.123357 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:04:37.124942 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.125913 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.128040 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.128350 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.128400 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.131290 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:04:37.131292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:04:37.131293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:04:37.131301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:04:37.133203 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:37.135275 (+ 2072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:37.135276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:37.135277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:37.135350 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:37.135699 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:37.135700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:37.135700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:37.135854 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:04:37.135911 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:37.136443 (+ 532us) server_negotiation.cc:299] Negotiation successful
0927 04:04:37.136544 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:04:37.136924 29849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.122189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:37451 (local address 127.17.184.195:59117)
0927 04:04:37.122383 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:37.122436 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 04:04:37.123172 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:04:37.123558 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:04:37.123562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:04:37.124506 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:04:37.124819 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.124827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.126507 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.126507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.127925 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.127928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.128504 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.128505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.128666 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.129373 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:04:37.129383 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:04:37.131162 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:04:37.134649 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:04:37.134660 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:04:37.134663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:04:37.135060 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:04:37.135435 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:04:37.135436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:04:37.135437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:04:37.135593 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:04:37.135948 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:04:37.135955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:04:37.136237 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 04:04:37.136377 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 04:04:37.136647 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260927 04:04:37.137746 29847 heartbeater.cc:347] Connected to a master server at 127.17.184.254:37451
I20260927 04:04:37.137850 29847 heartbeater.cc:464] Registering TS with master...
I20260927 04:04:37.138057 29847 heartbeater.cc:511] Master 127.17.184.254:37451 requested a full tablet report, sending...
I20260927 04:04:37.138495 29363 ts_manager.cc:194] Registered new tserver with Master: 04372450365d47398e7c7f4f56ba2f18 (127.17.184.195:34325)
I20260927 04:04:37.139027 29363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:59117
I20260927 04:04:37.146989 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:04:37.158399 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.148898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42866 (local address 127.17.184.254:37451)
0927 04:04:37.148944 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:37.148946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:37.149334 (+ 388us) server_negotiation.cc:407] Connection header received
0927 04:04:37.149442 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:37.149442 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:37.149454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:37.149484 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:04:37.150319 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.151129 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.152311 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.152617 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.152669 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.154605 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:04:37.154607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:04:37.154608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:04:37.154690 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:04:37.156882 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:37.157392 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:37.157393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:37.157393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:37.157471 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:04:37.157762 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:04:37.157763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:04:37.157763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:04:37.157928 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:04:37.158059 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:37.158196 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 04:04:37.158288 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:04:37.160761 29363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:04:37.163467 29363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:04:37.180047 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.168561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34325 (local address 127.0.0.1:58396)
0927 04:04:37.174190 (+ 5629us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:37.174259 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 04:04:37.174959 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:04:37.175151 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:04:37.175151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:04:37.175153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:04:37.175201 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.175202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.176545 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.176547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.178203 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.178207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.179672 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.179673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.179788 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.179813 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 04:04:37.179891 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 04:04:37.179952 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5568,"thread_start_us":106,"threads_started":1}
I20260927 04:04:37.180459 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.168898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.17.184.195:34325)
0927 04:04:37.169110 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:37.169116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:37.174680 (+ 5564us) server_negotiation.cc:407] Connection header received
0927 04:04:37.175060 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:37.175063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:37.175079 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:37.175143 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:04:37.175266 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.176433 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.178379 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.179449 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.179562 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.179631 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:37.180323 (+ 692us) server_negotiation.cc:299] Negotiation successful
0927 04:04:37.180374 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":91,"threads_started":1}
I20260927 04:04:37.181363 29781 tablet_service.cc:1467] Processing CreateTablet for tablet 59ab730ab6a745fbb3268d0d4fed7269 (DEFAULT_TABLE table=test-table [id=27b1534d8e8e4ff091c0ced510982527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:04:37.181787 29781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ab730ab6a745fbb3268d0d4fed7269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:04:37.181968 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.168619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:39557 (local address 127.0.0.1:59546)
0927 04:04:37.174505 (+ 5886us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:37.174515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:04:37.174583 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:04:37.174915 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:04:37.174920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:04:37.174923 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:04:37.175018 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.175020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.178440 (+ 3420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.178441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.179832 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.179836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.181706 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.181707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.181782 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.181787 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:04:37.181832 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 04:04:37.181880 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5870,"thread_start_us":87,"threads_started":1}
I20260927 04:04:37.182307 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.168906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.17.184.194:39557)
0927 04:04:37.174448 (+ 5542us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:37.174455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:37.174707 (+ 252us) server_negotiation.cc:407] Connection header received
0927 04:04:37.174751 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:37.174754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:37.174771 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:37.174851 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:04:37.176566 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.178101 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.179986 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.181499 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.181604 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.181663 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:37.182162 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 04:04:37.182214 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5493,"thread_start_us":91,"threads_started":1}
I20260927 04:04:37.183035 29647 tablet_service.cc:1467] Processing CreateTablet for tablet 59ab730ab6a745fbb3268d0d4fed7269 (DEFAULT_TABLE table=test-table [id=27b1534d8e8e4ff091c0ced510982527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:04:37.183387 29647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ab730ab6a745fbb3268d0d4fed7269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:04:37.186216 29865 tablet_bootstrap.cc:492] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: Bootstrap starting.
I20260927 04:04:37.187407 29865 tablet_bootstrap.cc:654] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: Neither blocks nor log segments found. Creating new log.
I20260927 04:04:37.187820 29865 log.cc:824] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: Log is configured to *not* fsync() on all Append() calls
I20260927 04:04:37.188668 29865 tablet_bootstrap.cc:492] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: No bootstrap required, opened a new log
I20260927 04:04:37.188751 29865 ts_tablet_manager.cc:1397] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:04:37.191293 29865 raft_consensus.cc:348] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.191491 29865 raft_consensus.cc:374] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:04:37.191529 29865 raft_consensus.cc:729] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04372450365d47398e7c7f4f56ba2f18, State: Initialized, Role: FOLLOWER
I20260927 04:04:37.191668 29865 consensus_queue.cc:260] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.191927 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.169182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:34909 (local address 127.0.0.1:54550)
0927 04:04:37.178243 (+ 9061us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:37.178256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:04:37.178312 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:04:37.187773 (+ 9461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:04:37.187775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:04:37.187779 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:04:37.187840 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.187842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.189216 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.189218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.190430 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.190434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.191587 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.191589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.191712 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.191718 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 04:04:37.191777 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 04:04:37.191830 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9039,"thread_start_us":68,"threads_started":1}
I20260927 04:04:37.191972 29865 ts_tablet_manager.cc:1428] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:04:37.192512 29847 heartbeater.cc:503] Master 127.17.184.254:37451 was elected leader, sending a full tablet report...
I20260927 04:04:37.192720 29864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.169273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.17.184.193:34909)
0927 04:04:37.187506 (+ 18233us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:37.187511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:37.187527 (+ 16us) server_negotiation.cc:407] Connection header received
0927 04:04:37.187567 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:37.187569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:37.187589 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:37.187682 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:04:37.187935 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.189083 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.190557 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.191392 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.191480 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.191541 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:37.192142 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 04:04:37.192630 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18187,"thread_start_us":64,"threads_started":1}
I20260927 04:04:37.193486 29509 tablet_service.cc:1467] Processing CreateTablet for tablet 59ab730ab6a745fbb3268d0d4fed7269 (DEFAULT_TABLE table=test-table [id=27b1534d8e8e4ff091c0ced510982527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:04:37.193872 29509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59ab730ab6a745fbb3268d0d4fed7269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:04:37.199330 29866 tablet_bootstrap.cc:492] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: Bootstrap starting.
I20260927 04:04:37.199416 29869 tablet_bootstrap.cc:492] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012: Bootstrap starting.
I20260927 04:04:37.200333 29869 tablet_bootstrap.cc:654] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012: Neither blocks nor log segments found. Creating new log.
I20260927 04:04:37.200619 29866 tablet_bootstrap.cc:654] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: Neither blocks nor log segments found. Creating new log.
I20260927 04:04:37.200829 29869 log.cc:824] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012: Log is configured to *not* fsync() on all Append() calls
I20260927 04:04:37.201009 29866 log.cc:824] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:04:37.201848 29869 tablet_bootstrap.cc:492] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012: No bootstrap required, opened a new log
I20260927 04:04:37.201879 29866 tablet_bootstrap.cc:492] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: No bootstrap required, opened a new log
I20260927 04:04:37.201931 29869 ts_tablet_manager.cc:1397] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:04:37.201943 29866 ts_tablet_manager.cc:1397] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:04:37.204200 29866 raft_consensus.cc:348] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.204423 29866 raft_consensus.cc:374] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:04:37.204459 29866 raft_consensus.cc:729] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171584f6ef73488cac6ef3f0cd824ea3, State: Initialized, Role: FOLLOWER
I20260927 04:04:37.204603 29866 consensus_queue.cc:260] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.204773 29869 raft_consensus.cc:348] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.204943 29869 raft_consensus.cc:374] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:04:37.204977 29869 raft_consensus.cc:729] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a599449ea14693a6b80486155a6012, State: Initialized, Role: FOLLOWER
I20260927 04:04:37.204924 29866 ts_tablet_manager.cc:1428] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:04:37.205219 29713 heartbeater.cc:503] Master 127.17.184.254:37451 was elected leader, sending a full tablet report...
I20260927 04:04:37.205413 29869 consensus_queue.cc:260] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.205724 29869 ts_tablet_manager.cc:1428] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260927 04:04:37.205844 29575 heartbeater.cc:503] Master 127.17.184.254:37451 was elected leader, sending a full tablet report...
Sep 27 04:04:37 dist-test-slave-zqrn krb5kdc[28887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481876, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:04:37.299830 29872 raft_consensus.cc:482] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260927 04:04:37.299880 29576 tablet.cc:2369] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:04:37.299991 29872 raft_consensus.cc:504] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.300405 29872 leader_election.cc:288] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04372450365d47398e7c7f4f56ba2f18 (127.17.184.195:34325), c0a599449ea14693a6b80486155a6012 (127.17.184.193:34909)
I20260927 04:04:37.305243 29715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.300539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34325 (local address 127.17.184.194:46719)
0927 04:04:37.300644 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:37.300655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:04:37.300696 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:04:37.300951 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:04:37.300952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:04:37.300954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:04:37.301031 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.301032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.302271 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.302272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.303601 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.303604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.304944 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.304945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.305027 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.305031 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:04:37.305077 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 04:04:37.305126 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":6}
I20260927 04:04:37.305568 29861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.300776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:46719 (local address 127.17.184.195:34325)
0927 04:04:37.300816 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:37.300818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:37.300825 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:04:37.300853 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:37.300853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:37.300862 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:37.300905 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:04:37.301102 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.301922 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.304206 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.304807 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.304881 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.304891 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:37.305455 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 04:04:37.305501 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:04:37.305876 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ab730ab6a745fbb3268d0d4fed7269" candidate_uuid: "171584f6ef73488cac6ef3f0cd824ea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04372450365d47398e7c7f4f56ba2f18" is_pre_election: true
I20260927 04:04:37.306093 29801 raft_consensus.cc:2454] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171584f6ef73488cac6ef3f0cd824ea3 in term 0.
I20260927 04:04:37.306375 29601 leader_election.cc:302] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04372450365d47398e7c7f4f56ba2f18, 171584f6ef73488cac6ef3f0cd824ea3; no voters:
I20260927 04:04:37.307049 29872 raft_consensus.cc:2781] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:04:37.307119 29872 raft_consensus.cc:482] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:04:37.307143 29872 raft_consensus.cc:3037] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:04:37.308272 29872 raft_consensus.cc:504] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.308429 29872 leader_election.cc:288] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [CANDIDATE]: Term 1 election: Requested vote from peers 04372450365d47398e7c7f4f56ba2f18 (127.17.184.195:34325), c0a599449ea14693a6b80486155a6012 (127.17.184.193:34909)
I20260927 04:04:37.308682 29801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ab730ab6a745fbb3268d0d4fed7269" candidate_uuid: "171584f6ef73488cac6ef3f0cd824ea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04372450365d47398e7c7f4f56ba2f18"
I20260927 04:04:37.308779 29801 raft_consensus.cc:3037] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:04:37.309695 29801 raft_consensus.cc:2454] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171584f6ef73488cac6ef3f0cd824ea3 in term 1.
I20260927 04:04:37.310070 29601 leader_election.cc:302] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04372450365d47398e7c7f4f56ba2f18, 171584f6ef73488cac6ef3f0cd824ea3; no voters:
I20260927 04:04:37.310215 29872 raft_consensus.cc:2781] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:04:37.310386 29872 raft_consensus.cc:686] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 1 LEADER]: Becoming Leader. State: Replica: 171584f6ef73488cac6ef3f0cd824ea3, State: Running, Role: LEADER
I20260927 04:04:37.310484 29872 consensus_queue.cc:237] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } }
I20260927 04:04:37.311446 29362 catalog_manager.cc:5495] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 171584f6ef73488cac6ef3f0cd824ea3 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "171584f6ef73488cac6ef3f0cd824ea3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171584f6ef73488cac6ef3f0cd824ea3" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 39557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 } health_report { overall_health: UNKNOWN } } }
I20260927 04:04:37.313067 29864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.301021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:40765 (local address 127.17.184.193:34909)
0927 04:04:37.301066 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:37.301068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:37.301325 (+ 257us) server_negotiation.cc:407] Connection header received
0927 04:04:37.301421 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:37.301422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:37.301432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:37.301460 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:04:37.301662 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.302376 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.306139 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.306812 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.306869 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.306880 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:37.312906 (+ 6026us) server_negotiation.cc:299] Negotiation successful
0927 04:04:37.312980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:04:37.313082 29874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.300645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:34909 (local address 127.17.184.194:40765)
0927 04:04:37.301263 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:37.301274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:04:37.301350 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:04:37.301519 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:04:37.301520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:04:37.301521 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:04:37.301597 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.301599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.303205 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.303206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.306001 (+ 2795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:04:37.306004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.312710 (+ 6706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.312712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:04:37.312811 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.312816 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:04:37.312871 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:04:37.312920 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":112,"threads_started":1}
I20260927 04:04:37.313655 29529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ab730ab6a745fbb3268d0d4fed7269" candidate_uuid: "171584f6ef73488cac6ef3f0cd824ea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a599449ea14693a6b80486155a6012" is_pre_election: true
I20260927 04:04:37.314319 29529 raft_consensus.cc:2454] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 171584f6ef73488cac6ef3f0cd824ea3 in term 0.
I20260927 04:04:37.314476 29528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59ab730ab6a745fbb3268d0d4fed7269" candidate_uuid: "171584f6ef73488cac6ef3f0cd824ea3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a599449ea14693a6b80486155a6012"
I20260927 04:04:37.314544 29528 raft_consensus.cc:3037] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:04:37.315541 29528 raft_consensus.cc:2454] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 171584f6ef73488cac6ef3f0cd824ea3 in term 1.
I20260927 04:04:37.323292 29862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.318518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59552 (local address 127.17.184.194:39557)
0927 04:04:37.318565 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:04:37.318568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:04:37.318576 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:04:37.318614 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:04:37.318615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:04:37.318637 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:04:37.318708 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:04:37.318971 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.319736 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.321784 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:04:37.322122 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:04:37.322201 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:04:37.322449 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:04:37.322601 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:04:37.322686 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:04:37.323170 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 04:04:37.323197 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:04:37.332333 29528 raft_consensus.cc:1261] T 59ab730ab6a745fbb3268d0d4fed7269 P c0a599449ea14693a6b80486155a6012 [term 1 FOLLOWER]: Refusing update from remote peer 171584f6ef73488cac6ef3f0cd824ea3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:04:37.332871 29875 consensus_queue.cc:1035] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0a599449ea14693a6b80486155a6012" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 34909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:04:37.335238 29801 raft_consensus.cc:1261] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 1 FOLLOWER]: Refusing update from remote peer 171584f6ef73488cac6ef3f0cd824ea3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:04:37.335644 29875 consensus_queue.cc:1035] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04372450365d47398e7c7f4f56ba2f18" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:04:37.340456 29878 mvcc.cc:205] Tried to move back new op lower bound from 7333813769549385728 to 7333813769464532992. Current Snapshot: MvccSnapshot[applied={T|T < 7333813769549385728}]
I20260927 04:04:37.340758 29881 mvcc.cc:205] Tried to move back new op lower bound from 7333813769549385728 to 7333813769464532992. Current Snapshot: MvccSnapshot[applied={T|T < 7333813769549385728}]
I20260927 04:04:37.342388 29879 mvcc.cc:205] Tried to move back new op lower bound from 7333813769549385728 to 7333813769464532992. Current Snapshot: MvccSnapshot[applied={T|T < 7333813769549385728}]
I20260927 04:04:37.346127 29362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:04:37.346249 29362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:04:37.347267 29362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6674dcaf3ad1471a859ce8a88f9a15b6: Sending DeleteTablet for 3 replicas of tablet 59ab730ab6a745fbb3268d0d4fed7269
I20260927 04:04:37.347999 29781 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ab730ab6a745fbb3268d0d4fed7269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:04:37 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:58396
I20260927 04:04:37.348335 29647 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ab730ab6a745fbb3268d0d4fed7269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:04:37 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:59546
I20260927 04:04:37.348575 29509 tablet_service.cc:1514] Processing DeleteTablet for tablet 59ab730ab6a745fbb3268d0d4fed7269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:04:37 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:54550
I20260927 04:04:37.349386 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 29444
I20260927 04:04:37.367048 29887 tablet_replica.cc:330] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: stopping tablet replica
I20260927 04:04:37.367282 29888 tablet_replica.cc:330] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: stopping tablet replica
I20260927 04:04:37.367492 29888 raft_consensus.cc:2229] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:04:37.367686 29887 raft_consensus.cc:2229] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:04:37.368779 29887 raft_consensus.cc:2258] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:04:37.367957 29888 raft_consensus.cc:2258] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:04:37.369637 29888 ts_tablet_manager.cc:1905] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:04:37.370131 29887 ts_tablet_manager.cc:1905] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:04:37.376636 29888 ts_tablet_manager.cc:1918] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:04:37.376938 29888 log.cc:1196] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/59ab730ab6a745fbb3268d0d4fed7269
I20260927 04:04:37.377318 29888 ts_tablet_manager.cc:1939] T 59ab730ab6a745fbb3268d0d4fed7269 P 171584f6ef73488cac6ef3f0cd824ea3: Deleting consensus metadata
I20260927 04:04:37.377928 29350 catalog_manager.cc:4841] TS 171584f6ef73488cac6ef3f0cd824ea3 (127.17.184.194:39557): tablet 59ab730ab6a745fbb3268d0d4fed7269 (table test-table [id=27b1534d8e8e4ff091c0ced510982527]) successfully deleted
I20260927 04:04:37.379565 29887 ts_tablet_manager.cc:1918] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:04:37.379735 29887 log.cc:1196] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/59ab730ab6a745fbb3268d0d4fed7269
I20260927 04:04:37.380491 29887 ts_tablet_manager.cc:1939] T 59ab730ab6a745fbb3268d0d4fed7269 P 04372450365d47398e7c7f4f56ba2f18: Deleting consensus metadata
I20260927 04:04:37.381060 29348 catalog_manager.cc:4841] TS 04372450365d47398e7c7f4f56ba2f18 (127.17.184.195:34325): tablet 59ab730ab6a745fbb3268d0d4fed7269 (table test-table [id=27b1534d8e8e4ff091c0ced510982527]) successfully deleted
W20260927 04:04:37.381446 29347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:34909 (error 108)
I20260927 04:04:37.382515 29859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:04:37.382348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:34909 (local address 127.0.0.1:54562)
0927 04:04:37.382425 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0927 04:04:37.382475 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:34909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 04:04:37.382791 29347 catalog_manager.cc:4584] TS c0a599449ea14693a6b80486155a6012 (127.17.184.193:34909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59ab730ab6a745fbb3268d0d4fed7269: Network error: Client connection negotiation failed: client connection to 127.17.184.193:34909: connect: Connection refused (error 111)
I20260927 04:04:37.383105 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 29581
I20260927 04:04:37.390355 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 29718
I20260927 04:04:37.395591 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 29331
I20260927 04:04:37.402225 18147 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 04:04:37.748172 18147 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 04:04:37.748353 18147 mini_ranger.cc:66] Stopping Ranger...
I20260927 04:04:38.094882 18147 mini_ranger.cc:68] Stopped Ranger
2026-09-27 04:04:38.095 UTC [28917] LOG: received smart shutdown request
2026-09-27 04:04:38.100 UTC [28917] LOG: background worker "logical replication launcher" (PID 28926) exited with exit code 1
2026-09-27 04:04:38.100 UTC [28921] LOG: shutting down
2026-09-27 04:04:38.390 UTC [28917] LOG: database system is shut down
2026-09-27T04:04:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48500 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:04:38 dist-test-slave-zqrn krb5kdc[29899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:04:38 dist-test-slave-zqrn krb5kdc[29899](info): set up 2 sockets
Sep 27 04:04:38 dist-test-slave-zqrn krb5kdc[29899](info): commencing operation
krb5kdc: starting...
W20260927 04:04:39.643065 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:04:39 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:04:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.184.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.184.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/rangeradmin_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/rangeradmin_127.17.184.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.184.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.184.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/rangerlookup_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/rangerlookup_127.17.184.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.184.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.184.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/HTTP_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/HTTP_127.17.184.212@KRBTEST.COM.keytab.
I20260927 04:04:39.831337 18147 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskJqOnmp/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/postgres -l logfile start
2026-09-27 04:04:42.632 UTC [29929] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 04:04:42.632 UTC [29929] LOG: listening on IPv4 address "127.17.184.212", port 55117
2026-09-27 04:04:42.636 UTC [29929] LOG: listening on Unix socket "/tmp/.s.PGSQL.55117"
2026-09-27 04:04:42.672 UTC [29932] LOG: database system was shut down at 2026-09-27 04:04:40 UTC
2026-09-27 04:04:42.676 UTC [29929] LOG: database system is ready to accept connections
I20260927 04:04:43.622722 18147 mini_postgres.cc:96] Postgres bound to 55117
2026-09-27 04:04:43.628 UTC [29940] FATAL: database "slave" does not exist
127.17.184.212:55117 - accepting connections
I20260927 04:04:43.629272 18147 mini_ranger.cc:161] Starting Ranger...
I20260927 04:04:43.643882 18147 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 04:04:44.187474 18147 mini_ranger.cc:87] Created ranger Postgres database
I20260927 04:04:44.187582 18147 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-admin
2026-09-27 04:04:44,465 [I] DB FLAVOR :POSTGRES
2026-09-27 04:04:44,465 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 04:04:44,466 [I] Checking connection..
2026-09-27 04:04:44,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 04:04:44,729 [I] Checking connection passed.
2026-09-27 04:04:44,730 [I] --------- Verifying version history table ---------
2026-09-27 04:04:44,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:04:44,997 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 04:04:44,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:04:45,292 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 04:04:45,292 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 04:04:45,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 04:04:45.530 UTC [30004] WARNING: there is no transaction in progress
2026-09-27 04:04:45,536 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 04:04:45,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 04:04:45,810 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 04:04:45,810 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 04:04:45,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 04:04:46,077 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 04:04:46,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zqrn.c.gcp-upstream.internal','N') ;"
2026-09-27 04:04:46,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 04:04:46,829 [I] Table x_portal_user does not exist in database ranger
2026-09-27 04:04:46,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 04:04:47,081 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 04:04:47,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 04:04:47,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 04:04:47,592 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 04:04:47,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 04:04:48.590 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.607 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.623 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.640 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.656 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.673 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.731 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.738 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.754 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.765 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.775 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.789 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.801 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.808 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.815 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:48.822 UTC [30112] WARNING: there is no transaction in progress
2026-09-27 04:04:49,524 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 04:04:49,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 04:04:49,773 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 04:04:49,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 04:04:50,017 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 04:04:50,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 04:04:50,283 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 04:04:50,283 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 04:04:50,533 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 04:04:50,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zqrn.c.gcp-upstream.internal';"
2026-09-27 04:04:51,030 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 04:04:51,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 04:04:51,267 [I] DB_PATCHES have already been applied
I20260927 04:04:51.274006 18147 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-admin:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskJqOnmp/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 04:04:51.274120 18147 mini_ranger.cc:193] Using host: 127.17.184.212
I20260927 04:04:51.277400 18147 mini_ranger.cc:227] Ranger admin URL: http://127.17.184.212:49905
Sep 27, 2026 4:04:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:04:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:04:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 4:04:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 4:04:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 4:04:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 4:04:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 04:04:52.416509 18147 mini_ranger.cc:160] Time spent starting Ranger: real 8.787s user 0.000s sys 0.009s
Sep 27, 2026 4:05:11 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 4:05:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 4:05:11 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 4:05:11 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 4:05:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 04:05:13.437729 18147 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.184.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.184.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/rangerkms_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.184.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/rangerkms_127.17.184.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.184.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.184.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.184.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/HTTP_127.17.184.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.184.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/HTTP_127.17.184.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 04:05:13 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481913, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 04:05:13.524084 18147 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 04:05:13.533087 18147 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 04:05:14.480713 18147 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 04:05:14.480808 18147 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms
I20260927 04:05:14.480829 18147 mini_ranger_kms.cc:227] Using postgres at 127.17.184.212:55117
2026-09-27 04:05:14,644 [I] DB FLAVOR :POSTGRES
2026-09-27 04:05:14,644 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 04:05:14,645 [I] Checking connection
2026-09-27 04:05:14,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 04:05:14,889 [I] connection success
2026-09-27 04:05:14,889 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 04:05:14,889 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 04:05:14,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 04:05:15,145 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 04:05:15,145 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 04:05:15,146 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 04:05:15,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.184.212:55117/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 04:05:15,405 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 04:05:15.879500 18147 mini_ranger_kms.cc:310] Created kms service
I20260927 04:05:16.065045 18147 mini_ranger_kms.cc:326] Created kudu user
I20260927 04:05:16.108791 18147 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 04:05:16 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481913, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:16.505306 18147 mini_ranger_kms.cc:384] Added ranger policy
I20260927 04:05:16.505527 18147 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms:/tmp/dist-test-taskJqOnmp/build/release/bin/postgresql.jar:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskJqOnmp/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 04:05:16.505549 18147 mini_ranger_kms.cc:240] Using host: 127.17.184.212
I20260927 04:05:16.509410 18147 mini_ranger_kms.cc:276] Ranger KMS PID: 30292
I20260927 04:05:16.509492 18147 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.184.212:46221
Sep 27, 2026 4:05:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 4:05:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 4:05:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 4:05:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 4:05:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 4:05:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 04:05:18.784798 18147 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.261s user 0.020s sys 0.126s
I20260927 04:05:18.784878 18147 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 04:05:18.784914 18147 mini_ranger_kms.cc:401] 127.17.184.212:46221/kms/v1/keys
Sep 27 04:05:19 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481919, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.184.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 04:05:19 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481919, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.184.212@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
Sep 27 04:05:19 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481919, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.184.212@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
Sep 27 04:05:22 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:05:22.306645 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44873
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44873
--ranger_config_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:46221/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:22.391304 30343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:22.391574 30343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:22.391614 30343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:22.392900 30343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:22.392977 30343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:22.393003 30343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:22.393035 30343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:22.393064 30343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:22.394749 30343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.184.212:46221/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44873
--ranger_config_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:22.395085 30343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:22.395334 30343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:22.400527 30348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:22.400496 30351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:22.400507 30349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:22.400926 30343 server_base.cc:1034] running on GCE node
I20260927 04:05:22.401218 30343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:22.401620 30343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:22.402812 30343 hybrid_clock.cc:648] HybridClock initialized: now 1790481922402801 us; error 37 us; skew 500 ppm
Sep 27 04:05:22 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481922, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:22.405822 30343 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:05:22.406352 30343 webserver.cc:460] Webserver started at http://127.17.184.254:46187/ using document root <none> and password file <none>
I20260927 04:05:22.406608 30343 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:22.406673 30343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:22.406829 30343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:05:22 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481922, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:22.463406 30343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "ac5401e99f06460eaaf450a7ff86b6ae"
format_stamp: "Formatted at 2026-09-27 04:05:22 on dist-test-slave-zqrn"
server_key: "54fabe225c5036012ff5b830195179b5"
server_key_iv: "9bd6138cce5dc2b4dbd152de3439f68c"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:22.463884 30343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "ac5401e99f06460eaaf450a7ff86b6ae"
format_stamp: "Formatted at 2026-09-27 04:05:22 on dist-test-slave-zqrn"
server_key: "54fabe225c5036012ff5b830195179b5"
server_key_iv: "9bd6138cce5dc2b4dbd152de3439f68c"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:22.465866 30343 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 04:05:22.478770 30358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:22.479033 30343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:22.479153 30343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "ac5401e99f06460eaaf450a7ff86b6ae"
format_stamp: "Formatted at 2026-09-27 04:05:22 on dist-test-slave-zqrn"
server_key: "54fabe225c5036012ff5b830195179b5"
server_key_iv: "9bd6138cce5dc2b4dbd152de3439f68c"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:22.479261 30343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:22.501631 30343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:22.504047 30343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:22.504235 30343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:22.509743 30343 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44873
I20260927 04:05:22.509824 30410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44873 every 8 connection(s)
I20260927 04:05:22.510229 30343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:05:22.511046 30411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:22.513520 30411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Bootstrap starting.
I20260927 04:05:22.513856 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 30343
I20260927 04:05:22.513976 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:05:22.514165 18147 external_mini_cluster.cc:1428] Setting key 7ed09408767a1c2b05df921a337b539f
I20260927 04:05:22.514329 30411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:22.514745 30411 log.cc:824] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:22.515602 30411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: No bootstrap required, opened a new log
I20260927 04:05:22.517522 30411 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } }
I20260927 04:05:22.517702 30411 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:22.517769 30411 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5401e99f06460eaaf450a7ff86b6ae, State: Initialized, Role: FOLLOWER
I20260927 04:05:22.517949 30411 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } }
I20260927 04:05:22.518052 30411 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:05:22.518111 30411 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:05:22.518193 30411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:22.519009 30411 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } }
I20260927 04:05:22.519124 30411 leader_election.cc:302] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac5401e99f06460eaaf450a7ff86b6ae; no voters:
I20260927 04:05:22.519341 30411 leader_election.cc:288] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:05:22.519404 30416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:22.519619 30416 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 1 LEADER]: Becoming Leader. State: Replica: ac5401e99f06460eaaf450a7ff86b6ae, State: Running, Role: LEADER
I20260927 04:05:22.519680 30411 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:22.520352 30416 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } }
I20260927 04:05:22.521006 30417 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } } }
I20260927 04:05:22.521147 30417 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:22.521333 30417 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac5401e99f06460eaaf450a7ff86b6ae. Latest consensus state: current_term: 1 leader_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } } }
I20260927 04:05:22.521405 30417 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:22.521049 30411 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 04:05:22.521484 30430 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:05:22.522008 30430 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 04:05:22 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:22.530653 30414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:22.514748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.17.184.254:44873)
0927 04:05:22.514989 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:22.514998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:22.515030 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:05:22.515564 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:22.515574 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:22.515659 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:22.515815 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:22.516667 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:22.518407 (+ 1740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:22.519638 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:22.520287 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:22.520396 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:22.524586 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:22.524649 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:22.524650 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:22.524791 (+ 141us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:22.528358 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:22.528875 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:22.528878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:22.528879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:22.528997 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:22.529266 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:22.529267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:22.529267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:22.529708 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:22.529890 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:22.529980 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 04:05:22.530421 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
I20260927 04:05:22.537127 30430 catalog_manager.cc:1347] Generated new cluster ID: 534f4a6145654c039c74e4e487d7fedb
I20260927 04:05:22.537194 30430 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:05:22.558526 30430 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:05:22.559399 30430 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:05:22.566171 30430 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Generated new TSK 0
I20260927 04:05:22.566457 30430 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:05:23.299171 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:46221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44873
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:23.413096 30456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:23.413347 30456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:23.413388 30456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:23.414681 30456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:23.414750 30456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:23.414808 30456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:05:23.416344 30456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.184.212:46221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:23.416586 30456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:23.416859 30456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:23.424329 30462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:23.424348 30464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:23.424904 30461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:23.425290 30456 server_base.cc:1034] running on GCE node
I20260927 04:05:23.425495 30456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:23.425765 30456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:23.426957 30456 hybrid_clock.cc:648] HybridClock initialized: now 1790481923426935 us; error 33 us; skew 500 ppm
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:23.430229 30456 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:05:23.430757 30456 webserver.cc:460] Webserver started at http://127.17.184.193:43621/ using document root <none> and password file <none>
I20260927 04:05:23.431047 30456 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:23.431119 30456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:23.431269 30456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:23.444514 30456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "219a4ae4e0729911668affda76246d90"
server_key_iv: "fdc04ea12ebba8ca5ec48877149b1b6b"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.444975 30456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "219a4ae4e0729911668affda76246d90"
server_key_iv: "fdc04ea12ebba8ca5ec48877149b1b6b"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.446843 30456 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 04:05:23.463081 30471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:23.463306 30456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:23.463402 30456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "219a4ae4e0729911668affda76246d90"
server_key_iv: "fdc04ea12ebba8ca5ec48877149b1b6b"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.463496 30456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:23.489113 30456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:23.494271 30456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:23.494498 30456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:23.494805 30456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:23.495231 30456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:23.495313 30456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:23.495405 30456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:23.495471 30456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:23.501772 30456 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:36887
I20260927 04:05:23.502262 30456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:05:23.505535 30586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:36887 every 8 connection(s)
I20260927 04:05:23.506548 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 30456
I20260927 04:05:23.506652 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:05:23.506826 18147 external_mini_cluster.cc:1428] Setting key 0bb060ceca58b33b4ca0d5f05c0e47ba
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
I20260927 04:05:23.535957 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:23.506256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:43981 (local address 127.17.184.254:44873)
0927 04:05:23.507033 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:23.507035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:23.507048 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:05:23.507523 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:23.507525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:23.507539 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:23.507603 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:23.508387 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:23.509193 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:23.512116 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:23.512453 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:23.512969 (+ 516us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:23.532031 (+ 19062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:23.532034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:23.532035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:23.532046 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:23.534052 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:23.534681 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:23.534683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:23.534683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:23.534751 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:23.535135 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:23.535136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:23.535137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:23.535295 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:23.535376 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:23.535746 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 04:05:23.535847 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260927 04:05:23.536144 30590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:23.506239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44873 (local address 127.17.184.193:43981)
0927 04:05:23.506488 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:23.506533 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:23.507353 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:23.507751 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:23.507756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:23.507999 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:23.508279 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:23.508284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:23.509293 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:23.509294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:23.511985 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:23.511987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:23.512565 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:23.512566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:23.512721 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:23.513325 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:23.513335 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:23.523371 (+ 10036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:23.534173 (+ 10802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:23.534188 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:23.534191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:23.534574 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:23.534837 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:23.534839 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:23.534839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:23.535038 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:23.535371 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:23.535379 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:23.535558 (+ 179us) client_negotiation.cc:769] Sending connection context
0927 04:05:23.535666 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:05:23.535837 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260927 04:05:23.536726 30587 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44873
I20260927 04:05:23.536840 30587 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:23.537057 30587 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:23.537645 30375 ts_manager.cc:194] Registered new tserver with Master: db59bfe156c542a4b4ce2ca7f21b9a2d (127.17.184.193:36887)
I20260927 04:05:23.538704 30375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:43981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:05:23.543294 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:46221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44873
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 04:05:23.664999 30593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:23.665334 30593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:23.665416 30593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:23.667194 30593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:23.667296 30593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:23.667430 30593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:05:23.669525 30593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.184.212:46221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:23.669957 30593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:23.670279 30593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:23.677035 30601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:23.677042 30600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:23.677441 30603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:23.678315 30593 server_base.cc:1034] running on GCE node
I20260927 04:05:23.678532 30593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:23.678793 30593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:23.679960 30593 hybrid_clock.cc:648] HybridClock initialized: now 1790481923679941 us; error 48 us; skew 500 ppm
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:23.683548 30593 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:05:23.684113 30593 webserver.cc:460] Webserver started at http://127.17.184.194:33105/ using document root <none> and password file <none>
I20260927 04:05:23.684432 30593 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:23.684558 30593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:23.684748 30593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:23.701980 30593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "2e63f11f8cb747cc94575ee34ff10992"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "172f6533da096aeb1ca2781a282b8fd2"
server_key_iv: "1b0ab1048bd5d1157cf9b3e86942dc75"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.702391 30593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "2e63f11f8cb747cc94575ee34ff10992"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "172f6533da096aeb1ca2781a282b8fd2"
server_key_iv: "1b0ab1048bd5d1157cf9b3e86942dc75"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.704445 30593 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260927 04:05:23.725754 30610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:23.726089 30593 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260927 04:05:23.726194 30593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "2e63f11f8cb747cc94575ee34ff10992"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "172f6533da096aeb1ca2781a282b8fd2"
server_key_iv: "1b0ab1048bd5d1157cf9b3e86942dc75"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.726291 30593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:23.742969 30593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:23.745400 30593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:23.745574 30593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:23.745855 30593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:23.746260 30593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:23.746304 30593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:23.746367 30593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:23.746397 30593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:23.753737 30593 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:44015
I20260927 04:05:23.754124 30593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:05:23.754602 30724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:44015 every 8 connection(s)
I20260927 04:05:23.759913 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 30593
I20260927 04:05:23.760011 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:05:23.760196 18147 external_mini_cluster.cc:1428] Setting key 3d054f19f02340c1368852300201a5f8
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:23.765699 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:23.754972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:39141 (local address 127.17.184.254:44873)
0927 04:05:23.755059 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:23.755062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:23.755288 (+ 226us) server_negotiation.cc:407] Connection header received
0927 04:05:23.755819 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:23.755820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:23.755837 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:23.755899 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:23.756800 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:23.757597 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:23.758945 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:23.759208 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:23.759243 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:23.762223 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:23.762225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:23.762226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:23.762233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:23.764071 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:23.764572 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:23.764573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:23.764573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:23.764633 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:23.764910 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:23.764910 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:23.764911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:23.765054 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:23.765100 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:23.765503 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 04:05:23.765604 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 04:05:23.766450 30727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:23.754884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44873 (local address 127.17.184.194:39141)
0927 04:05:23.755154 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:23.755197 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:23.755712 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:23.755972 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:23.755976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:23.756400 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:23.756683 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:23.756690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:23.757725 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:23.757726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:23.758821 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:23.758824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:23.759325 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:23.759325 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:23.759417 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:23.760479 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:23.760491 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:23.762122 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:23.764166 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:23.764172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:23.764175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:23.764462 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:23.764706 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:23.764706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:23.764707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:23.764835 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:23.765152 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:23.765158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:23.765338 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 04:05:23.766045 (+ 707us) client_negotiation.cc:240] Negotiation successful
0927 04:05:23.766213 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":85,"threads_started":1}
I20260927 04:05:23.766947 30725 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44873
I20260927 04:05:23.767047 30725 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:23.767238 30725 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:23.767601 30375 ts_manager.cc:194] Registered new tserver with Master: 2e63f11f8cb747cc94575ee34ff10992 (127.17.184.194:44015)
I20260927 04:05:23.768059 30375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:39141
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:05:23.794401 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:46221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44873
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:23.917500 30730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:23.917785 30730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:23.917860 30730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:23.919536 30730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:23.919608 30730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:23.919654 30730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:05:23.921440 30730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.184.212:46221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:23.921705 30730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:23.921949 30730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:23.928071 30736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:23.928072 30738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:23.928120 30735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:23.928577 30730 server_base.cc:1034] running on GCE node
I20260927 04:05:23.928835 30730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:23.929100 30730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:23.930279 30730 hybrid_clock.cc:648] HybridClock initialized: now 1790481923930255 us; error 37 us; skew 500 ppm
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:23.933288 30730 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:05:23.933809 30730 webserver.cc:460] Webserver started at http://127.17.184.195:36707/ using document root <none> and password file <none>
I20260927 04:05:23.934062 30730 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:23.934136 30730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:23.934314 30730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 04:05:23 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:23.955279 30730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "027fdbaffe6449b5953ff1abeedb8624"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "953afe2404d5b1d52ee8de6ac0573655"
server_key_iv: "662e5f2006eed16967ba7437b6089cec"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.955900 30730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "027fdbaffe6449b5953ff1abeedb8624"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "953afe2404d5b1d52ee8de6ac0573655"
server_key_iv: "662e5f2006eed16967ba7437b6089cec"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.959012 30730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260927 04:05:23.975509 30745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:23.975778 30730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:05:23.975874 30730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "027fdbaffe6449b5953ff1abeedb8624"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "953afe2404d5b1d52ee8de6ac0573655"
server_key_iv: "662e5f2006eed16967ba7437b6089cec"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:23.975973 30730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:24.017450 30730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:24.020287 30730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:24.020502 30730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:24.020856 30730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:24.021302 30730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:24.021441 30730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.021517 30730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:24.021574 30730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.029083 30730 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:46863
I20260927 04:05:24.029605 30730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:05:24.030992 30858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:46863 every 8 connection(s)
I20260927 04:05:24.035890 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 30730
I20260927 04:05:24.035998 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:05:24.036190 18147 external_mini_cluster.cc:1428] Setting key bf10d40e2eff9bff04c2f440ea7d1c7f
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481923, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:24.042464 30589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.030876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:42767 (local address 127.17.184.254:44873)
0927 04:05:24.030949 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:24.030951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:24.031412 (+ 461us) server_negotiation.cc:407] Connection header received
0927 04:05:24.031973 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:24.031975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:24.031989 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:24.032032 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:24.032642 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.033436 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.034563 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.034879 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.034965 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.039260 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:24.039262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:24.039262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:24.039271 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:24.040758 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.041246 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.041247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.041247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.041309 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.041606 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.041607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.041607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.041807 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:24.041858 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:24.042266 (+ 408us) server_negotiation.cc:299] Negotiation successful
0927 04:05:24.042366 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 04:05:24.043054 30861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.030524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44873 (local address 127.17.184.195:42767)
0927 04:05:24.031277 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:24.031312 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:24.031849 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:24.032114 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:24.032118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:24.032342 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:24.032543 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:24.032547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.033556 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.033558 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:24.034457 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:24.034460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.035084 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.035085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:24.035211 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.035822 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:24.035834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:24.039162 (+ 3328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:24.040862 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:24.040870 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:24.040873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:24.041153 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:24.041383 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:24.041384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:24.041384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:24.041525 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:24.041923 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:24.041930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:24.042100 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 04:05:24.042618 (+ 518us) client_negotiation.cc:240] Negotiation successful
0927 04:05:24.042795 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":81,"threads_started":1}
I20260927 04:05:24.043541 30859 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44873
I20260927 04:05:24.043664 30859 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.043846 30859 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:24.044240 30375 ts_manager.cc:194] Registered new tserver with Master: 027fdbaffe6449b5953ff1abeedb8624 (127.17.184.195:46863)
I20260927 04:05:24.044682 30375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:42767
I20260927 04:05:24.048840 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:05:24.049557 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 30456
I20260927 04:05:24.056002 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 30593
I20260927 04:05:24.062121 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 30730
I20260927 04:05:24.067920 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 30343
I20260927 04:05:24.073060 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44873
--webserver_interface=127.17.184.254
--webserver_port=46187
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44873
--ranger_config_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:46221/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.184.212:46221/kms with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:24.153609 30862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:24.153811 30862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:24.153832 30862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:24.154937 30862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:24.154974 30862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:24.154987 30862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:24.155000 30862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:24.155012 30862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:24.156490 30862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.184.212:46221/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44873
--ranger_config_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=46187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:24.156670 30862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:24.156833 30862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:24.161697 30868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:24.161689 30870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:24.161720 30862 server_base.cc:1034] running on GCE node
W20260927 04:05:24.161667 30867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:24.162074 30862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:24.162364 30862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:24.163530 30862 hybrid_clock.cc:648] HybridClock initialized: now 1790481924163509 us; error 45 us; skew 500 ppm
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:24.166246 30862 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:05:24.166806 30862 webserver.cc:460] Webserver started at http://127.17.184.254:46187/ using document root <none> and password file <none>
I20260927 04:05:24.167084 30862 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:24.167168 30862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:24.231822 30862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:24.232599 30878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.232796 30862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:24.232893 30862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "ac5401e99f06460eaaf450a7ff86b6ae"
format_stamp: "Formatted at 2026-09-27 04:05:22 on dist-test-slave-zqrn"
server_key: "54fabe225c5036012ff5b830195179b5"
server_key_iv: "9bd6138cce5dc2b4dbd152de3439f68c"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:24.233284 30862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:24.249531 30862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:24.251892 30862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:24.252007 30862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:24.256217 30862 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44873
I20260927 04:05:24.256255 30930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44873 every 8 connection(s)
I20260927 04:05:24.256618 30862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:05:24.257793 30931 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:05:24.258440 30931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Bootstrap starting.
I20260927 04:05:24.259409 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 30862
I20260927 04:05:24.259729 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:46221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:36887
--local_ip_for_outbound_sockets=127.17.184.193
--tserver_master_addrs=127.17.184.254:44873
--webserver_port=43621
--webserver_interface=127.17.184.193
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.184.212:46221/kms with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:24.269160 30931 log.cc:824] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:24.271780 30931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:05:24.272164 30931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Bootstrap complete.
I20260927 04:05:24.275923 30931 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } }
I20260927 04:05:24.276275 30931 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5401e99f06460eaaf450a7ff86b6ae, State: Initialized, Role: FOLLOWER
I20260927 04:05:24.276451 30931 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } }
I20260927 04:05:24.276520 30931 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:05:24.276566 30931 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:05:24.276616 30931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:05:24.278047 30931 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } }
I20260927 04:05:24.278173 30931 leader_election.cc:302] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac5401e99f06460eaaf450a7ff86b6ae; no voters:
I20260927 04:05:24.278345 30931 leader_election.cc:288] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 04:05:24.278422 30935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 2 FOLLOWER]: Leader election won for term 2
I20260927 04:05:24.278595 30935 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [term 2 LEADER]: Becoming Leader. State: Replica: ac5401e99f06460eaaf450a7ff86b6ae, State: Running, Role: LEADER
I20260927 04:05:24.278605 30931 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:24.278769 30935 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } }
I20260927 04:05:24.279044 30936 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } } }
I20260927 04:05:24.279119 30936 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:24.279368 30937 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac5401e99f06460eaaf450a7ff86b6ae. Latest consensus state: current_term: 2 leader_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5401e99f06460eaaf450a7ff86b6ae" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44873 } } }
I20260927 04:05:24.279484 30940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:05:24.279490 30937 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:24.280022 30940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:05:24.280170 30940 catalog_manager.cc:1259] Loaded cluster ID: 534f4a6145654c039c74e4e487d7fedb
I20260927 04:05:24.280233 30940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:05:24.280431 30931 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 04:05:24.283815 30940 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:05:24.284477 30940 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Loaded TSK: 0
I20260927 04:05:24.284802 30940 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 04:05:24.387562 30933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:24.387892 30933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:24.387974 30933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:24.389734 30933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:24.389797 30933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:24.389842 30933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:05:24.392318 30933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.184.212:46221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:36887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=43621
--tserver_master_addrs=127.17.184.254:44873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:24.392592 30933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:24.392897 30933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:24.398788 30971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:24.398756 30969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:24.399421 30933 server_base.cc:1034] running on GCE node
W20260927 04:05:24.399544 30968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:24.399866 30933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:24.400154 30933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:24.403479 30933 hybrid_clock.cc:648] HybridClock initialized: now 1790481924403440 us; error 57 us; skew 500 ppm
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:24.407126 30933 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:05:24.407740 30933 webserver.cc:460] Webserver started at http://127.17.184.193:43621/ using document root <none> and password file <none>
I20260927 04:05:24.408058 30933 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:24.408183 30933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:24.433137 30933 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:24.435374 30979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.435542 30933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 04:05:24.435636 30933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "219a4ae4e0729911668affda76246d90"
server_key_iv: "fdc04ea12ebba8ca5ec48877149b1b6b"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:24.436139 30933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:24.450668 30933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:24.453177 30933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:24.453342 30933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:24.453539 30933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:24.453902 30933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:24.453951 30933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.453992 30933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:24.454028 30933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.461064 30933 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:36887
I20260927 04:05:24.461530 30933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:05:24.462805 31092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:36887 every 8 connection(s)
I20260927 04:05:24.470029 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 30933
I20260927 04:05:24.470328 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:46221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:44015
--local_ip_for_outbound_sockets=127.17.184.194
--tserver_master_addrs=127.17.184.254:44873
--webserver_port=33105
--webserver_interface=127.17.184.194
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.184.212:46221/kms with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:24.478520 31095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.463536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44873 (local address 127.17.184.193:34851)
0927 04:05:24.464267 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:24.464312 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:24.464922 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:24.465728 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:24.465731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:24.465958 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:24.466238 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:24.466244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.467880 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.467881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:24.469190 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:24.469192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.472057 (+ 2865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.472058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:24.472195 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.472845 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:24.472855 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:24.474498 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:24.476726 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:24.476733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:24.476736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:24.477077 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:24.477348 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:24.477348 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:24.477349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:24.477462 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:24.477889 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:24.477895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:24.478046 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 04:05:24.478147 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 04:05:24.478294 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":93,"threads_started":1}
I20260927 04:05:24.479182 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.463913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:34851 (local address 127.17.184.254:44873)
0927 04:05:24.465141 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:24.465150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:24.465186 (+ 36us) server_negotiation.cc:407] Connection header received
0927 04:05:24.465326 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:24.465333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:24.465391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:24.465579 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:24.466361 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.467769 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.469275 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.471836 (+ 2561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.471936 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.474628 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:24.474632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:24.474632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:24.474643 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:24.476627 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.477171 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.477175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.477175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.477278 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.477528 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.477529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.477529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.477789 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:24.477854 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:24.478793 (+ 939us) server_negotiation.cc:299] Negotiation successful
0927 04:05:24.478928 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1173,"thread_start_us":79,"threads_started":1}
I20260927 04:05:24.479774 31093 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44873
I20260927 04:05:24.479862 31093 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.480080 31093 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
W20260927 04:05:24.480988 31093 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:44873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 04:05:24.481072 31093 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.481137 31093 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:24.481419 31093 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.481493 31093 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
W20260927 04:05:24.481755 31093 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:05:24.584688 31097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:24.584939 31097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:24.584976 31097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:24.586201 31097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:24.586253 31097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:24.586290 31097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:05:24.587929 31097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.184.212:46221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:44015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=33105
--tserver_master_addrs=127.17.184.254:44873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:24.588209 31097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:24.588460 31097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:24.593792 31105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:24.593781 31103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:24.594182 31102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:24.594619 31097 server_base.cc:1034] running on GCE node
I20260927 04:05:24.594851 31097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:24.595182 31097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:24.596417 31097 hybrid_clock.cc:648] HybridClock initialized: now 1790481924596393 us; error 48 us; skew 500 ppm
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:24.599130 31097 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:05:24.599653 31097 webserver.cc:460] Webserver started at http://127.17.184.194:33105/ using document root <none> and password file <none>
I20260927 04:05:24.599896 31097 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:24.599948 31097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:24.619407 31097 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:24.623428 31113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.623692 31097 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260927 04:05:24.623803 31097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "2e63f11f8cb747cc94575ee34ff10992"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "172f6533da096aeb1ca2781a282b8fd2"
server_key_iv: "1b0ab1048bd5d1157cf9b3e86942dc75"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:24.624221 31097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:24.643951 31097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:24.646526 31097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:24.646732 31097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:24.647015 31097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:24.647392 31097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:24.647439 31097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.647470 31097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:24.647490 31097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.654963 31097 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:44015
I20260927 04:05:24.655133 31226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:44015 every 8 connection(s)
I20260927 04:05:24.655444 31097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:24.665552 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 31097
I20260927 04:05:24.665876 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.184.212:46221/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:46863
--local_ip_for_outbound_sockets=127.17.184.195
--tserver_master_addrs=127.17.184.254:44873
--webserver_port=36707
--webserver_interface=127.17.184.195
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.184.212:46221/kms with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:24.666535 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.656521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:38683 (local address 127.17.184.254:44873)
0927 04:05:24.656574 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:24.656576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:24.656674 (+ 98us) server_negotiation.cc:407] Connection header received
0927 04:05:24.657363 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:24.657364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:24.657376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:24.657421 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:24.658162 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.658656 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.660183 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.660386 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.660499 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.663188 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:24.663189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:24.663189 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:24.663198 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:24.664906 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.665382 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.665384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.665385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.665449 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.665752 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.665754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.665754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.665897 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:24.665941 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:24.666344 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 04:05:24.666437 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:05:24.667107 31229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.656227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44873 (local address 127.17.184.194:38683)
0927 04:05:24.656443 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:24.656578 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:24.657241 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:24.657515 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:24.657520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:24.657765 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:24.658049 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:24.658055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.658758 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.658759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:24.660084 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:24.660087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.660532 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.660533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:24.660670 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.661287 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:24.661296 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:24.663017 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:24.665005 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:24.665011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:24.665013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:24.665239 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:24.665527 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:24.665528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:24.665528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:24.665666 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:24.665986 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:24.665992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:24.666162 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 04:05:24.666667 (+ 505us) client_negotiation.cc:240] Negotiation successful
0927 04:05:24.666834 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260927 04:05:24.667574 31227 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44873
I20260927 04:05:24.667660 31227 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.667843 31227 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
W20260927 04:05:24.668402 31227 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:44873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 04:05:24.668462 31227 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.668517 31227 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:24.668816 31227 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.668880 31227 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
W20260927 04:05:24.669162 31227 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:05:24.788470 31230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:24.788782 31230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:24.788822 31230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:24.790514 31230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:24.790580 31230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:24.790623 31230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:05:24.792848 31230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.184.212:46221/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:46863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=36707
--tserver_master_addrs=127.17.184.254:44873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:24.793103 31230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:24.793391 31230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:24.798739 31239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:24.798728 31237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:24.798986 31236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:24.798987 31230 server_base.cc:1034] running on GCE node
I20260927 04:05:24.799239 31230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:24.799481 31230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:24.800699 31230 hybrid_clock.cc:648] HybridClock initialized: now 1790481924800685 us; error 33 us; skew 500 ppm
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:24.803558 31230 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:05:24.804000 31230 webserver.cc:460] Webserver started at http://127.17.184.195:36707/ using document root <none> and password file <none>
I20260927 04:05:24.804226 31230 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:24.804308 31230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:24.830367 31230 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260927 04:05:24.831830 31247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.835078 31230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 04:05:24.835189 31230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "027fdbaffe6449b5953ff1abeedb8624"
format_stamp: "Formatted at 2026-09-27 04:05:23 on dist-test-slave-zqrn"
server_key: "953afe2404d5b1d52ee8de6ac0573655"
server_key_iv: "662e5f2006eed16967ba7437b6089cec"
server_key_version: "kuduclusterkey@0"
I20260927 04:05:24.835749 31230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:24.844990 31230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:24.847424 31230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:24.847658 31230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:24.847923 31230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:24.848301 31230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:24.848368 31230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.848419 31230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:24.848454 31230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:24.860244 31230 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:46863
I20260927 04:05:24.860339 31360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:46863 every 8 connection(s)
I20260927 04:05:24.860684 31230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:05:24.864040 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 31230
Sep 27 04:05:24 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481924, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:24.872526 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.864461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.17.184.254:44873)
0927 04:05:24.864771 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:24.864774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:24.864780 (+ 6us) server_negotiation.cc:407] Connection header received
0927 04:05:24.864798 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:24.864798 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:24.864808 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:24.864864 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:24.865640 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.866386 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.867506 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.867819 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.867905 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.869127 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:24.869128 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:24.869129 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:24.869136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:24.871183 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.871581 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.871583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.871583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.871668 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.871899 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.871900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.871900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.872045 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:24.872229 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:24.872308 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:05:24.872400 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":79,"threads_started":1}
I20260927 04:05:24.872715 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.861479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:60649 (local address 127.17.184.254:44873)
0927 04:05:24.861571 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:24.861574 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:24.861795 (+ 221us) server_negotiation.cc:407] Connection header received
0927 04:05:24.862497 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:24.862499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:24.862581 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:24.862654 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:24.863494 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.864167 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.865941 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.866135 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.866247 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.869375 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:24.869377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:24.869377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:24.869385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:24.871058 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.871592 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.871593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.871593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.871690 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:24.871969 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:24.871970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:24.871971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:24.872083 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:24.872165 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:24.872534 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 04:05:24.872609 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 04:05:24.873145 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:24.861455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44873 (local address 127.17.184.195:60649)
0927 04:05:24.861639 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:24.861690 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:24.862352 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:24.862758 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:24.862762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:24.863057 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:24.863367 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:24.863374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.864606 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.864608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:24.865856 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:24.865859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:24.866272 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:24.866274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:24.866392 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:24.867200 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:24.867210 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:24.869279 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:24.871153 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:24.871159 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:24.871161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:24.871496 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:24.871780 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:24.871781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:24.871782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:24.871914 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:24.872191 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:24.872197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:24.872372 (+ 175us) client_negotiation.cc:769] Sending connection context
0927 04:05:24.872759 (+ 387us) client_negotiation.cc:240] Negotiation successful
0927 04:05:24.872917 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260927 04:05:24.873769 31361 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44873
I20260927 04:05:24.873880 31361 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.874082 31361 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
W20260927 04:05:24.874863 31361 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:44873 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 04:05:24.874974 31361 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.875053 31361 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:24.875435 31361 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:24.875532 31361 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
W20260927 04:05:24.875838 31361 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 04:05:25 dist-test-slave-zqrn krb5kdc[29899](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790481925, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:25.481949 31093 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:25.482108 31093 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:25.482547 30895 ts_manager.cc:194] Registered new tserver with Master: db59bfe156c542a4b4ce2ca7f21b9a2d (127.17.184.193:36887)
I20260927 04:05:25.483618 30895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:34851
I20260927 04:05:25.669355 31227 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:25.669525 31227 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:25.669888 30895 ts_manager.cc:194] Registered new tserver with Master: 2e63f11f8cb747cc94575ee34ff10992 (127.17.184.194:44015)
I20260927 04:05:25.670421 30895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:38683
Sep 27 04:05:25 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481925, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
I20260927 04:05:25.876029 31361 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:25.876197 31361 heartbeater.cc:511] Master 127.17.184.254:44873 requested a full tablet report, sending...
I20260927 04:05:25.876533 30895 ts_manager.cc:194] Registered new tserver with Master: 027fdbaffe6449b5953ff1abeedb8624 (127.17.184.195:46863)
I20260927 04:05:25.877039 30895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:60649
I20260927 04:05:25.878330 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:05:25.891037 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:25.883159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.17.184.254:44873)
0927 04:05:25.883547 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:25.883549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:25.883558 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:05:25.883581 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:25.883581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:25.883594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:25.884314 (+ 720us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:25.884355 (+ 41us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.885093 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.886088 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.886375 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.887853 (+ 1478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:25.887902 (+ 49us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:25.887903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:25.887903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:25.887908 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:25.889560 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:25.889944 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:25.889945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:25.889945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:25.889998 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:25.890175 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:25.890175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:25.890176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:25.890296 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:25.890799 (+ 503us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:25.890840 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 04:05:25.890937 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":77,"threads_started":1}
I20260927 04:05:25.893184 30895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:05:25.895648 30895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:05:25.904732 31388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:25.898664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:44015 (local address 127.0.0.1:38146)
0927 04:05:25.899702 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:25.899773 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:25.899843 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:25.900051 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:25.900054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:25.900056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:25.900130 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:25.900132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.901422 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.901423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:25.902739 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:25.902741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.904436 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.904437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:25.904523 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:25.904539 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:05:25.904588 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 04:05:25.904638 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"mutex_wait_us":18,"thread_start_us":93,"threads_started":1}
I20260927 04:05:25.905323 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:25.899321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.17.184.194:44015)
0927 04:05:25.899430 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:25.899434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:25.899851 (+ 417us) server_negotiation.cc:407] Connection header received
0927 04:05:25.899919 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:25.899922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:25.899939 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:25.900210 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:25.900277 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.901323 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.903501 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.904324 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.905091 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:25.905161 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:25.905200 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 04:05:25.905245 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":46,"threads_started":1}
I20260927 04:05:25.906193 31161 tablet_service.cc:1467] Processing CreateTablet for tablet d6a449e8e8384c4fbdc4affb01e80915 (DEFAULT_TABLE table=test-table [id=268362720d334800bdec61f4bdd6e8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:25.906564 31161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a449e8e8384c4fbdc4affb01e80915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:25.908562 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:25.899183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.17.184.193:36887)
0927 04:05:25.899327 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:25.899332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:25.900008 (+ 676us) server_negotiation.cc:407] Connection header received
0927 04:05:25.900117 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:25.900120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:25.900134 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:25.900180 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:25.900385 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.901637 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.904444 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.905251 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.905292 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:25.905339 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:25.908432 (+ 3093us) server_negotiation.cc:299] Negotiation successful
0927 04:05:25.908485 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260927 04:05:25.908788 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:25.898664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:36887 (local address 127.0.0.1:40182)
0927 04:05:25.899958 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:25.899966 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:25.900044 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:25.900269 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:25.900270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:25.900272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:25.900312 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:25.900314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.901716 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.901717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:25.904033 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:25.904035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.905511 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.905511 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:25.905750 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:25.905754 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:05:25.908671 (+ 2917us) client_negotiation.cc:240] Negotiation successful
0927 04:05:25.908716 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"mutex_wait_us":5,"thread_start_us":79,"threads_started":1}
I20260927 04:05:25.909446 31027 tablet_service.cc:1467] Processing CreateTablet for tablet d6a449e8e8384c4fbdc4affb01e80915 (DEFAULT_TABLE table=test-table [id=268362720d334800bdec61f4bdd6e8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:25.909777 31027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a449e8e8384c4fbdc4affb01e80915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:25.913313 31395 tablet_bootstrap.cc:492] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d: Bootstrap starting.
I20260927 04:05:25.914026 31395 tablet_bootstrap.cc:654] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:25.915118 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:25.899028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.17.184.195:46863)
0927 04:05:25.899639 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:25.899643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:25.904181 (+ 4538us) server_negotiation.cc:407] Connection header received
0927 04:05:25.904279 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:25.904282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:25.904298 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:25.904348 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:25.905838 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.907228 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.913921 (+ 6693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.914662 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.914712 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:25.914762 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:25.914988 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 04:05:25.915043 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":67,"threads_started":1}
I20260927 04:05:25.915335 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:25.898921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:46863 (local address 127.0.0.1:57590)
0927 04:05:25.904127 (+ 5206us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:25.904137 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:25.904206 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:25.905392 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:25.905392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:25.905395 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:25.905435 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:25.905436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.907315 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.907316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:25.908320 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:25.908322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:25.914823 (+ 6501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:25.914824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:25.914884 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:25.914888 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:05:25.915220 (+ 332us) client_negotiation.cc:240] Negotiation successful
0927 04:05:25.915268 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5188,"thread_start_us":80,"threads_started":1}
I20260927 04:05:25.915958 31295 tablet_service.cc:1467] Processing CreateTablet for tablet d6a449e8e8384c4fbdc4affb01e80915 (DEFAULT_TABLE table=test-table [id=268362720d334800bdec61f4bdd6e8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:25.916270 31295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6a449e8e8384c4fbdc4affb01e80915. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:25.919042 31394 tablet_bootstrap.cc:492] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992: Bootstrap starting.
I20260927 04:05:25.919725 31394 tablet_bootstrap.cc:654] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:25.920042 31394 log.cc:824] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:25.920478 31396 tablet_bootstrap.cc:492] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Bootstrap starting.
I20260927 04:05:25.921041 31396 tablet_bootstrap.cc:654] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:25.921298 31396 log.cc:824] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:25.921945 31396 tablet_bootstrap.cc:492] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: No bootstrap required, opened a new log
I20260927 04:05:25.922017 31396 ts_tablet_manager.cc:1397] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:05:25.924367 31396 raft_consensus.cc:348] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:25.924532 31396 raft_consensus.cc:374] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:25.924571 31396 raft_consensus.cc:729] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 027fdbaffe6449b5953ff1abeedb8624, State: Initialized, Role: FOLLOWER
I20260927 04:05:25.924677 31396 consensus_queue.cc:260] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:25.924911 31396 ts_tablet_manager.cc:1428] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:05:25.925567 31361 heartbeater.cc:503] Master 127.17.184.254:44873 was elected leader, sending a full tablet report...
I20260927 04:05:25.925945 31395 log.cc:824] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:25.927336 31395 tablet_bootstrap.cc:492] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d: No bootstrap required, opened a new log
I20260927 04:05:25.927438 31395 ts_tablet_manager.cc:1397] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20260927 04:05:25.930879 31395 raft_consensus.cc:348] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:25.931104 31395 raft_consensus.cc:374] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:25.931145 31395 raft_consensus.cc:729] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db59bfe156c542a4b4ce2ca7f21b9a2d, State: Initialized, Role: FOLLOWER
I20260927 04:05:25.931281 31395 consensus_queue.cc:260] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:25.931891 31395 ts_tablet_manager.cc:1428] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 04:05:25.932035 31093 heartbeater.cc:503] Master 127.17.184.254:44873 was elected leader, sending a full tablet report...
I20260927 04:05:25.933662 31394 tablet_bootstrap.cc:492] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992: No bootstrap required, opened a new log
I20260927 04:05:25.933727 31394 ts_tablet_manager.cc:1397] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.000s
I20260927 04:05:25.936093 31394 raft_consensus.cc:348] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:25.936280 31394 raft_consensus.cc:374] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:25.936332 31394 raft_consensus.cc:729] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e63f11f8cb747cc94575ee34ff10992, State: Initialized, Role: FOLLOWER
I20260927 04:05:25.936499 31394 consensus_queue.cc:260] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:25.936790 31394 ts_tablet_manager.cc:1428] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:05:25.936908 31227 heartbeater.cc:503] Master 127.17.184.254:44873 was elected leader, sending a full tablet report...
W20260927 04:05:25.965245 31094 tablet.cc:2369] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 04:05:26 dist-test-slave-zqrn krb5kdc[29899](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481925, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for HTTP/127.17.184.212@KRBTEST.COM
W20260927 04:05:26.111910 31362 tablet.cc:2369] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:05:26.156777 31228 tablet.cc:2369] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:05:26.248332 31399 raft_consensus.cc:482] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:26.248490 31399 raft_consensus.cc:504] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:26.248930 31399 leader_election.cc:288] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db59bfe156c542a4b4ce2ca7f21b9a2d (127.17.184.193:36887), 2e63f11f8cb747cc94575ee34ff10992 (127.17.184.194:44015)
I20260927 04:05:26.260540 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:26.249375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:36171 (local address 127.17.184.193:36887)
0927 04:05:26.249493 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:26.249495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:26.253877 (+ 4382us) server_negotiation.cc:407] Connection header received
0927 04:05:26.254007 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:26.254009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:26.254019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:26.254050 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:26.254435 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.255271 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.259547 (+ 4276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.260112 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.260167 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:26.260178 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:26.260422 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 04:05:26.260468 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 04:05:26.261041 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:26.249095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:36887 (local address 127.17.184.195:36171)
0927 04:05:26.253771 (+ 4676us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:26.253787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:26.253917 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:26.254126 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:26.254127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:26.254129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:26.254209 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:26.254211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.255803 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.255804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:26.257036 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:26.257039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.260260 (+ 3221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.260261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:26.260330 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:26.260334 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:05:26.260889 (+ 555us) client_negotiation.cc:240] Negotiation successful
0927 04:05:26.260936 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4651,"spinlock_wait_cycles":12672,"thread_start_us":111,"threads_started":1}
I20260927 04:05:26.261574 31047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a449e8e8384c4fbdc4affb01e80915" candidate_uuid: "027fdbaffe6449b5953ff1abeedb8624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" is_pre_election: true
I20260927 04:05:26.261768 31047 raft_consensus.cc:2454] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 027fdbaffe6449b5953ff1abeedb8624 in term 0.
I20260927 04:05:26.262025 31248 leader_election.cc:302] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027fdbaffe6449b5953ff1abeedb8624, db59bfe156c542a4b4ce2ca7f21b9a2d; no voters:
I20260927 04:05:26.262166 31399 raft_consensus.cc:2781] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:05:26.262218 31399 raft_consensus.cc:482] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:05:26.262248 31399 raft_consensus.cc:3037] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:26.262606 31393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:26.249318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:58457 (local address 127.17.184.194:44015)
0927 04:05:26.249421 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:26.249424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:26.257210 (+ 7786us) server_negotiation.cc:407] Connection header received
0927 04:05:26.257305 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:26.257307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:26.257317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:26.257346 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:26.257531 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.258187 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.261488 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.262024 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.262073 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:26.262081 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:26.262503 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 04:05:26.262545 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 04:05:26.263105 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:26.249096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:44015 (local address 127.17.184.195:58457)
0927 04:05:26.257153 (+ 8057us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:26.257164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:26.257237 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:26.257403 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:26.257404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:26.257405 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:26.257464 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:26.257465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.258293 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.258294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:26.259432 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:26.259435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.262342 (+ 2907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.262343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:26.262419 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:26.262423 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:05:26.262976 (+ 553us) client_negotiation.cc:240] Negotiation successful
0927 04:05:26.263022 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8037,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260927 04:05:26.263223 31399 raft_consensus.cc:504] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:26.263374 31399 leader_election.cc:288] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [CANDIDATE]: Term 1 election: Requested vote from peers db59bfe156c542a4b4ce2ca7f21b9a2d (127.17.184.193:36887), 2e63f11f8cb747cc94575ee34ff10992 (127.17.184.194:44015)
I20260927 04:05:26.263535 31181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a449e8e8384c4fbdc4affb01e80915" candidate_uuid: "027fdbaffe6449b5953ff1abeedb8624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e63f11f8cb747cc94575ee34ff10992" is_pre_election: true
I20260927 04:05:26.263712 31181 raft_consensus.cc:2454] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 027fdbaffe6449b5953ff1abeedb8624 in term 0.
I20260927 04:05:26.263871 31180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a449e8e8384c4fbdc4affb01e80915" candidate_uuid: "027fdbaffe6449b5953ff1abeedb8624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e63f11f8cb747cc94575ee34ff10992"
I20260927 04:05:26.263950 31180 raft_consensus.cc:3037] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:26.264829 31180 raft_consensus.cc:2454] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 027fdbaffe6449b5953ff1abeedb8624 in term 1.
I20260927 04:05:26.265120 31251 leader_election.cc:302] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 027fdbaffe6449b5953ff1abeedb8624, 2e63f11f8cb747cc94575ee34ff10992; no voters:
I20260927 04:05:26.265358 31047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6a449e8e8384c4fbdc4affb01e80915" candidate_uuid: "027fdbaffe6449b5953ff1abeedb8624" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d"
I20260927 04:05:26.265447 31047 raft_consensus.cc:3037] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:26.266288 31047 raft_consensus.cc:2454] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 027fdbaffe6449b5953ff1abeedb8624 in term 1.
I20260927 04:05:26.266494 31399 raft_consensus.cc:2781] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:26.266659 31399 raft_consensus.cc:686] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 1 LEADER]: Becoming Leader. State: Replica: 027fdbaffe6449b5953ff1abeedb8624, State: Running, Role: LEADER
I20260927 04:05:26.266754 31399 consensus_queue.cc:237] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } }
I20260927 04:05:26.267598 30895 catalog_manager.cc:5495] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 reported cstate change: term changed from 0 to 1, leader changed from <none> to 027fdbaffe6449b5953ff1abeedb8624 (127.17.184.195). New cstate: current_term: 1 leader_uuid: "027fdbaffe6449b5953ff1abeedb8624" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "027fdbaffe6449b5953ff1abeedb8624" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46863 } health_report { overall_health: HEALTHY } } }
I20260927 04:05:26.348413 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:26.345225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57602 (local address 127.17.184.195:46863)
0927 04:05:26.345271 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:26.345273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:26.345281 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:05:26.345333 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:26.345334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:26.345352 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:26.345416 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:05:26.345577 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.346305 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.347419 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:26.347734 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:26.347811 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:26.347988 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:05:26.348141 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:05:26.348234 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:26.348280 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 04:05:26.348305 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:05:26.351029 31047 raft_consensus.cc:1261] T d6a449e8e8384c4fbdc4affb01e80915 P db59bfe156c542a4b4ce2ca7f21b9a2d [term 1 FOLLOWER]: Refusing update from remote peer 027fdbaffe6449b5953ff1abeedb8624: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:26.351038 31180 raft_consensus.cc:1261] T d6a449e8e8384c4fbdc4affb01e80915 P 2e63f11f8cb747cc94575ee34ff10992 [term 1 FOLLOWER]: Refusing update from remote peer 027fdbaffe6449b5953ff1abeedb8624: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:26.351389 31399 consensus_queue.cc:1035] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db59bfe156c542a4b4ce2ca7f21b9a2d" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 36887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:26.351594 31399 consensus_queue.cc:1035] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e63f11f8cb747cc94575ee34ff10992" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 44015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:26.356598 31417 mvcc.cc:205] Tried to move back new op lower bound from 7333813970330877952 to 7333813969989201920. Current Snapshot: MvccSnapshot[applied={T|T < 7333813970330877952}]
I20260927 04:05:26.356809 31418 mvcc.cc:205] Tried to move back new op lower bound from 7333813970330877952 to 7333813969989201920. Current Snapshot: MvccSnapshot[applied={T|T < 7333813970330877952}]
I20260927 04:05:26.359212 30894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:05:26.359301 30894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:05:26.359984 30894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac5401e99f06460eaaf450a7ff86b6ae: Sending DeleteTablet for 3 replicas of tablet d6a449e8e8384c4fbdc4affb01e80915
I20260927 04:05:26.360314 31027 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a449e8e8384c4fbdc4affb01e80915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:26 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:40182
I20260927 04:05:26.360430 31161 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a449e8e8384c4fbdc4affb01e80915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:26 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:38146
I20260927 04:05:26.360607 31295 tablet_service.cc:1514] Processing DeleteTablet for tablet d6a449e8e8384c4fbdc4affb01e80915 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:26 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:57590
I20260927 04:05:26.360821 31429 tablet_replica.cc:330] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: stopping tablet replica
I20260927 04:05:26.360953 31429 raft_consensus.cc:2229] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:05:26.361016 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 30933
I20260927 04:05:26.361176 31429 raft_consensus.cc:2258] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:26.363292 31429 ts_tablet_manager.cc:1905] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:26.365213 31429 ts_tablet_manager.cc:1918] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:05:26.365307 31429 log.cc:1196] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/d6a449e8e8384c4fbdc4affb01e80915
I20260927 04:05:26.365615 31429 ts_tablet_manager.cc:1939] T d6a449e8e8384c4fbdc4affb01e80915 P 027fdbaffe6449b5953ff1abeedb8624: Deleting consensus metadata
I20260927 04:05:26.366037 30880 catalog_manager.cc:4841] TS 027fdbaffe6449b5953ff1abeedb8624 (127.17.184.195:46863): tablet d6a449e8e8384c4fbdc4affb01e80915 (table test-table [id=268362720d334800bdec61f4bdd6e8ee]) successfully deleted
W20260927 04:05:26.371048 30879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:36887 (error 108)
I20260927 04:05:26.372004 31389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:26.371865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:36887 (local address 127.0.0.1:40184)
0927 04:05:26.371913 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:26.371956 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:36887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 04:05:26.372005 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 31097
W20260927 04:05:26.372133 30879 catalog_manager.cc:4584] TS db59bfe156c542a4b4ce2ca7f21b9a2d (127.17.184.193:36887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d6a449e8e8384c4fbdc4affb01e80915: Network error: Client connection negotiation failed: client connection to 127.17.184.193:36887: connect: Connection refused (error 111)
W20260927 04:05:26.378466 31389 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:05:26.378183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:44015 (local address 127.0.0.1:38150)
0927 04:05:26.378232 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:26.378245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:26.378293 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:26.378397 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.194:44015: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":23}
I20260927 04:05:26.378716 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 31230
I20260927 04:05:26.383702 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 30862
I20260927 04:05:26.392601 18147 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 04:05:26.737273 18147 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 04:05:26.737370 18147 mini_ranger.cc:66] Stopping Ranger...
I20260927 04:05:27.082285 18147 mini_ranger.cc:68] Stopped Ranger
2026-09-27 04:05:27.082 UTC [29929] LOG: received smart shutdown request
2026-09-27 04:05:27.086 UTC [29929] LOG: background worker "logical replication launcher" (PID 29938) exited with exit code 1
2026-09-27 04:05:27.086 UTC [29933] LOG: shutting down
2026-09-27 04:05:27.275 UTC [29929] LOG: database system is shut down
2026-09-27T04:05:27Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48874 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:05:27 dist-test-slave-zqrn krb5kdc[31439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:05:27 dist-test-slave-zqrn krb5kdc[31439](info): set up 2 sockets
Sep 27 04:05:27 dist-test-slave-zqrn krb5kdc[31439](info): commencing operation
krb5kdc: starting...
W20260927 04:05:28.455000 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:05:28 dist-test-slave-zqrn krb5kdc[31439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:05:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:05:28.573865 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:36321
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:36321,127.17.184.253:37421,127.17.184.252:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:28.658402 31452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:28.658607 31452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:28.658630 31452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:28.659860 31452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:28.659906 31452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:28.659924 31452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:28.659974 31452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:28.660012 31452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:28.661494 31452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:36321,127.17.184.253:37421,127.17.184.252:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:36321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:28.661746 31452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:28.662003 31452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:28.666867 31457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:28.667009 31460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:28.666922 31458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:28.666949 31452 server_base.cc:1034] running on GCE node
I20260927 04:05:28.667299 31452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:28.667543 31452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:28.668684 31452 hybrid_clock.cc:648] HybridClock initialized: now 1790481928668673 us; error 28 us; skew 500 ppm
Sep 27 04:05:28 dist-test-slave-zqrn krb5kdc[31439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:28.671166 31452 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:05:28.671535 31452 webserver.cc:460] Webserver started at http://127.17.184.254:41973/ using document root <none> and password file <none>
I20260927 04:05:28.671708 31452 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:28.671747 31452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:28.671839 31452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:28.672830 31452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "d8547b9858534eb8b435bafbfdab07c9"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "f4e189524e71212538bbd5703668e4fd"
server_key_iv: "258053da5e13845e1716eaad328b7828"
server_key_version: "encryptionkey@0"
I20260927 04:05:28.673128 31452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "d8547b9858534eb8b435bafbfdab07c9"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "f4e189524e71212538bbd5703668e4fd"
server_key_iv: "258053da5e13845e1716eaad328b7828"
server_key_version: "encryptionkey@0"
I20260927 04:05:28.674386 31452 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:05:28.675196 31467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:28.675374 31452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:28.675474 31452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "d8547b9858534eb8b435bafbfdab07c9"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "f4e189524e71212538bbd5703668e4fd"
server_key_iv: "258053da5e13845e1716eaad328b7828"
server_key_version: "encryptionkey@0"
I20260927 04:05:28.675560 31452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:28.693048 31452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:28.695289 31452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:28.695456 31452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:28.699465 31452 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:36321
I20260927 04:05:28.699523 31519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:36321 every 8 connection(s)
I20260927 04:05:28.699883 31452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:05:28.700752 31520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:28.702324 31520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 04:05:28 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:28.708842 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 31452
I20260927 04:05:28.708967 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:05:28.709229 18147 external_mini_cluster.cc:1428] Setting key decba378645b0b0f1291ff5a1c42ced7
I20260927 04:05:28.712430 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.703175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.17.184.254:36321)
0927 04:05:28.703295 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:28.703304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:28.703332 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:05:28.703805 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:28.703807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:28.703823 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:28.703900 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:28.704442 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.705259 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.706184 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.706510 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.706593 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:28.708884 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:28.708892 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:28.708892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:28.708899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:28.710782 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:28.711215 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:28.711218 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:28.711219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:28.711297 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:28.711538 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:28.711539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:28.711540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:28.711752 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:28.711846 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:28.712038 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 04:05:28.712167 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260927 04:05:28.712430 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.703023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:36321 (local address 127.0.0.1:60778)
0927 04:05:28.703189 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.703226 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:28.703674 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:28.703926 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:28.703929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:28.704207 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:28.704358 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:28.704362 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.705357 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.705358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:28.706077 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:28.706079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.706622 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.706624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:28.706703 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:28.707264 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:28.707272 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:28.708760 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:28.710881 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:28.710886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:28.710888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:28.711118 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:28.711374 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:28.711374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:28.711375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:28.711464 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:28.711825 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:28.711830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:28.711888 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 04:05:28.711993 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 04:05:28.712230 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":99,"thread_start_us":66,"threads_started":1}
I20260927 04:05:28.713397 31520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:05:28.713783 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.713660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51844)
0927 04:05:28.713706 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.713754 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 04:05:28.713914 31470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
I20260927 04:05:28.714339 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.714248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51860)
0927 04:05:28.714293 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.714316 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 04:05:28.714442 31520 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:37421: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.17.184.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab.
Entry for principal kudu/127.17.184.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab.
I20260927 04:05:28.734557 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.253:37421
--webserver_interface=127.17.184.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab
--principal=kudu/127.17.184.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:36321,127.17.184.253:37421,127.17.184.252:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:28.763688 31520 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } attempt: 1
I20260927 04:05:28.764227 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.764047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51870)
0927 04:05:28.764132 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.764185 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 04:05:28.764616 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.764522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51872)
0927 04:05:28.764567 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.764596 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 04:05:28.764839 31520 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:37421: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
W20260927 04:05:28.816332 31527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:28.816515 31527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:28.816536 31527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:28.817567 31527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:28.817600 31527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:28.817613 31527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:28.817625 31527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:28.817637 31527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:28.819118 31527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:36321,127.17.184.253:37421,127.17.184.252:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab
--principal=kudu/127.17.184.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.253:37421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.184.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:28.819501 31527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:28.819682 31527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:28.824260 31532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:28.824239 31533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:28.824270 31527 server_base.cc:1034] running on GCE node
W20260927 04:05:28.824254 31535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:28.824739 31527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:28.824968 31527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:28.826109 31527 hybrid_clock.cc:648] HybridClock initialized: now 1790481928826100 us; error 28 us; skew 500 ppm
Sep 27 04:05:28 dist-test-slave-zqrn krb5kdc[31439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:28.828572 31527 init.cc:377] Logged in from keytab as kudu/127.17.184.253@KRBTEST.COM (short username kudu)
I20260927 04:05:28.829038 31527 webserver.cc:460] Webserver started at http://127.17.184.253:35041/ using document root <none> and password file <none>
I20260927 04:05:28.829257 31527 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:28.829300 31527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:28.829460 31527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:28.830447 31527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/data/instance:
uuid: "539b934ea8024dd5bcab30a8ba987fbf"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "b40d04f480aa65fa5a60b47cea22ddc2"
server_key_iv: "d18642c2196d4668f18e45ff209f36b8"
server_key_version: "encryptionkey@0"
I20260927 04:05:28.830782 31527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/wal/instance:
uuid: "539b934ea8024dd5bcab30a8ba987fbf"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "b40d04f480aa65fa5a60b47cea22ddc2"
server_key_iv: "d18642c2196d4668f18e45ff209f36b8"
server_key_version: "encryptionkey@0"
I20260927 04:05:28.832576 31527 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260927 04:05:28.832983 31520 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } attempt: 2
I20260927 04:05:28.833400 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.833257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51882)
0927 04:05:28.833317 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.833360 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 04:05:28.833788 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.833684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51884)
0927 04:05:28.833736 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.833765 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 04:05:28.833791 31542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260927 04:05:28.833899 31520 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:37421: Network error: Client connection negotiation failed: client connection to 127.17.184.253:37421: connect: Connection refused (error 111)
I20260927 04:05:28.833976 31527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:05:28.834064 31527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/wal
uuid: "539b934ea8024dd5bcab30a8ba987fbf"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "b40d04f480aa65fa5a60b47cea22ddc2"
server_key_iv: "d18642c2196d4668f18e45ff209f36b8"
server_key_version: "encryptionkey@0"
I20260927 04:05:28.834143 31527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:28.859275 31527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:28.861717 31527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:28.861831 31527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:28.865782 31527 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.253:37421
I20260927 04:05:28.865834 31594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.253:37421 every 8 connection(s)
I20260927 04:05:28.866194 31527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
I20260927 04:05:28.867197 31595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:28.868754 31595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:05:28.869472 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 31527
I20260927 04:05:28.869604 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-1/wal/instance
I20260927 04:05:28.869786 18147 external_mini_cluster.cc:1428] Setting key 9e272edeaa804fd0704a9e56c008f7e8
Sep 27 04:05:28 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.252@KRBTEST.COM; defaulting to no policy
I20260927 04:05:28.878096 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.869479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.17.184.254:36321)
0927 04:05:28.869549 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:28.869552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:28.869742 (+ 190us) server_negotiation.cc:407] Connection header received
0927 04:05:28.870297 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:28.870298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:28.870314 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:28.870392 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:28.871017 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.871515 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.872536 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.872759 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.872848 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:28.875239 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:28.875241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:28.875241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:28.875250 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:28.876610 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:28.877075 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:28.877076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:28.877077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:28.877141 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:28.877400 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:28.877401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:28.877402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:28.877523 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:28.877625 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:28.877903 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 04:05:28.877992 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 04:05:28.878317 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.869437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:36321 (local address 127.0.0.1:60788)
0927 04:05:28.869615 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.869657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:28.870193 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:28.870460 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:28.870464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:28.870700 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:28.870935 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:28.870940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.871614 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.871616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:28.872443 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:28.872445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.872855 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.872857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:28.872999 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:28.873502 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:28.873510 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:28.875143 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:28.876748 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:28.876754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:28.876756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:28.877002 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:28.877216 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:28.877216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:28.877217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:28.877330 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:28.877617 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:28.877624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:28.877767 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 04:05:28.877905 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 04:05:28.878082 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260927 04:05:28.878806 31595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.252@KRBTEST.COM" created.
Sep 27 04:05:28 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:05:28.886428 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.879168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51892 (local address 127.17.184.253:37421)
0927 04:05:28.879289 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:28.879294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:28.879311 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:05:28.879340 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:28.879342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:28.879352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:28.879412 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:28.879820 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.880514 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.881490 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.881793 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.881851 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:28.883740 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:28.883746 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:28.883746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:28.883752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:28.885265 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:28.885667 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:28.885670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:28.885671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:28.885724 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:28.885920 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:28.885920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:28.885921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:28.886043 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:28.886102 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:28.886273 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 04:05:28.886353 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260927 04:05:28.886571 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.879106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51892)
0927 04:05:28.879152 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.879165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:28.879207 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:28.879425 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:28.879426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:28.879572 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:28.879649 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:28.879692 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.880607 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.880609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:28.881398 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:28.881400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.881869 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.881870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:28.881943 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:28.882371 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:28.882376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:28.883649 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:28.885353 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:28.885354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:28.885354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:28.885580 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:28.885793 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:28.885794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:28.885795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:28.885854 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:28.886112 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:28.886113 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:28.886163 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 04:05:28.886251 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 04:05:28.886414 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":60}
I20260927 04:05:28.887194 31595 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:05:28.887567 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.887438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56582)
0927 04:05:28.887493 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.887547 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:05:28.887703 31543 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
I20260927 04:05:28.888111 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.888021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56588)
0927 04:05:28.888063 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.888088 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 04:05:28.888218 31595 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:46227: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab.
Entry for principal kudu/127.17.184.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab.
I20260927 04:05:28.897763 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.252:46227
--webserver_interface=127.17.184.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab
--principal=kudu/127.17.184.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:33341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:36321,127.17.184.253:37421,127.17.184.252:46227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:28.925040 31520 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } attempt: 3
Sep 27 04:05:28 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:05:28.932417 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.925465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.17.184.253:37421)
0927 04:05:28.925516 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:28.925519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:28.925531 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:05:28.925587 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:28.925588 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:28.925599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:28.925672 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:28.925993 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.926498 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.927354 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.927587 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.927674 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:28.929707 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:28.929709 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:28.929709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:28.929717 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:28.931198 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:28.931602 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:28.931604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:28.931605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:28.931662 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:28.931880 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:28.931881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:28.931881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:28.931994 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:28.932075 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:28.932233 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 04:05:28.932319 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:05:28.932418 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.925395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51900)
0927 04:05:28.925459 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.925470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:28.925525 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:28.925677 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:28.925678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:28.925840 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:28.925901 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:28.925903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.926677 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.926679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:28.927252 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:28.927255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:28.927674 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:28.927675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:28.927756 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:28.928230 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:28.928234 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:28.929620 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:28.931332 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:28.931333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:28.931334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:28.931509 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:28.931731 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:28.931732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:28.931732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:28.931791 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:28.932070 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:28.932071 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:28.932117 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 04:05:28.932208 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 04:05:28.932317 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:05:28.932780 31520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:05:28.933144 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.933028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56604)
0927 04:05:28.933092 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.933119 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 04:05:28.933504 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.933424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56616)
0927 04:05:28.933467 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.933489 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 04:05:28.933672 31520 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:46227: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
I20260927 04:05:28.937444 31595 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } attempt: 1
I20260927 04:05:28.937887 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.937730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56628)
0927 04:05:28.937804 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.937846 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 04:05:28.938283 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.938161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56638)
0927 04:05:28.938216 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.938251 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 04:05:28.938447 31595 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:46227: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
I20260927 04:05:28.964861 31520 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } attempt: 1
I20260927 04:05:28.965444 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.965298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56650)
0927 04:05:28.965368 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.965412 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 04:05:28.965847 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:28.965756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56666)
0927 04:05:28.965804 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:28.965829 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:05:28.965957 31520 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:46227: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
W20260927 04:05:28.985044 31602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:28.985253 31602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:28.985275 31602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:28.986398 31602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:28.986439 31602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:28.986454 31602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:28.986466 31602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:28.986501 31602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:28.988065 31602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:36321,127.17.184.253:37421,127.17.184.252:46227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab
--principal=kudu/127.17.184.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.252:46227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.184.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:28.988333 31602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:28.988580 31602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:28.993441 31610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:28.993544 31607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:28.993741 31602 server_base.cc:1034] running on GCE node
W20260927 04:05:28.993544 31608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:28.994093 31602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:28.994410 31602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:28.995631 31602 hybrid_clock.cc:648] HybridClock initialized: now 1790481928995595 us; error 54 us; skew 500 ppm
Sep 27 04:05:28 dist-test-slave-zqrn krb5kdc[31439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:28.998432 31602 init.cc:377] Logged in from keytab as kudu/127.17.184.252@KRBTEST.COM (short username kudu)
I20260927 04:05:28.999027 31602 webserver.cc:460] Webserver started at http://127.17.184.252:32835/ using document root <none> and password file <none>
I20260927 04:05:28.999296 31602 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:28.999372 31602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:28.999532 31602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:29.000705 31602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/data/instance:
uuid: "b279712e1dfc42f29a6deb6961cb31a9"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "f96e3a45105ba098d5c29e55bcd08b33"
server_key_iv: "4aa39e832462334cb51d49de56c019cf"
server_key_version: "encryptionkey@0"
I20260927 04:05:29.001089 31602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/wal/instance:
uuid: "b279712e1dfc42f29a6deb6961cb31a9"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "f96e3a45105ba098d5c29e55bcd08b33"
server_key_iv: "4aa39e832462334cb51d49de56c019cf"
server_key_version: "encryptionkey@0"
I20260927 04:05:29.002590 31602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:05:29.003535 31617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:29.003722 31602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:05:29.003840 31602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/wal
uuid: "b279712e1dfc42f29a6deb6961cb31a9"
format_stamp: "Formatted at 2026-09-27 04:05:28 on dist-test-slave-zqrn"
server_key: "f96e3a45105ba098d5c29e55bcd08b33"
server_key_iv: "4aa39e832462334cb51d49de56c019cf"
server_key_version: "encryptionkey@0"
I20260927 04:05:29.003925 31602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:29.006628 31595 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } attempt: 2
I20260927 04:05:29.007123 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.006984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56676)
0927 04:05:29.007047 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:29.007094 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:05:29.007488 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.007389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56692)
0927 04:05:29.007436 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:29.007461 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 04:05:29.007579 31595 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:46227: Network error: Client connection negotiation failed: client connection to 127.17.184.252:46227: connect: Connection refused (error 111)
I20260927 04:05:29.025501 31602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:29.028101 31602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:29.028285 31602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:29.032984 31602 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.252:46227
I20260927 04:05:29.033020 31669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.252:46227 every 8 connection(s)
I20260927 04:05:29.033404 31602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
I20260927 04:05:29.034235 31670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:29.036526 31670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:05:29.041112 31520 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } attempt: 2
Sep 27 04:05:29 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:29.043215 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 31602
I20260927 04:05:29.043318 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790481773036938-18147-0/minicluster-data/master-2/wal/instance
I20260927 04:05:29.043499 18147 external_mini_cluster.cc:1428] Setting key d344106f3a718ab2ffe8b47f96faa119
Sep 27 04:05:29 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:05:29.046861 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.037349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.17.184.254:36321)
0927 04:05:29.037438 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.037441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.037691 (+ 250us) server_negotiation.cc:407] Connection header received
0927 04:05:29.038174 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.038176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.038188 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.038250 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.038875 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.039436 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.040448 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.040666 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.040756 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.043272 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.043274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.043274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.043281 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.045329 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.045786 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.045787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.045787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.045844 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.046149 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.046150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.046151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.046287 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.046354 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.046687 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.046774 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:05:29.047330 31672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.037302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:36321 (local address 127.0.0.1:60794)
0927 04:05:29.037508 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:29.037544 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:29.038051 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:29.038324 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:29.038327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:29.038574 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:29.038771 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.038775 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.039559 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.039560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.040344 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.040346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.040765 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.040766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.040885 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.041526 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:29.041536 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:29.043183 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:29.045415 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.045421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.045423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.045706 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.045921 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.045921 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.045922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.046065 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.046359 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:29.046364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:29.046517 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 04:05:29.046985 (+ 468us) client_negotiation.cc:240] Negotiation successful
0927 04:05:29.047151 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":105,"threads_started":1}
I20260927 04:05:29.048038 31670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:05:29.049765 31522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.041392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56702)
0927 04:05:29.041489 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:29.041501 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:29.041547 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:29.041766 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:29.041767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:29.041909 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:29.041949 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.041951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.042783 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.042784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.043610 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.043613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.044566 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.044567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.044634 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.045067 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:29.045071 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:29.046471 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:29.048567 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.048568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.048569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.048815 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.049050 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.049051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.049051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.049119 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.049465 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:29.049466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:29.049507 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 04:05:29.049558 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:05:29.049668 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
Sep 27 04:05:29 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:05:29.050241 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.041451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.17.184.252:46227)
0927 04:05:29.041605 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.041610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.041628 (+ 18us) server_negotiation.cc:407] Connection header received
0927 04:05:29.041656 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.041658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.041674 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.041736 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.042030 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.042688 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.043717 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.044200 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.044272 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.046552 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.046559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.046560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.046568 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.048464 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.048913 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.048917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.048917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.048977 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.049199 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.049200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.049200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.049376 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.050014 (+ 638us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.050062 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.050139 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":100,"threads_started":1}
Sep 27 04:05:29 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:05:29.052722 31520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9: Bootstrap starting.
I20260927 04:05:29.053396 31520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:29.053691 31520 log.cc:824] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:29.054447 31520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9: No bootstrap required, opened a new log
I20260927 04:05:29.055580 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.043872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56712 (local address 127.17.184.252:46227)
0927 04:05:29.044227 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.044229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.044236 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:05:29.044253 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.044253 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.044263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.044322 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.044967 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.045727 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.047042 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.047363 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.047429 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.050415 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.050416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.050417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.050423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.052913 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.053836 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.053837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.053838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.053901 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.054137 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.054138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.054139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.055089 (+ 950us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.055263 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.055352 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.055435 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":100,"threads_started":1}
I20260927 04:05:29.056036 31672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.048296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:37421 (local address 127.0.0.1:51910)
0927 04:05:29.048336 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:29.048346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:29.048380 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:29.048498 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:29.048498 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:29.048636 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:29.048662 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.048663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.049392 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.049394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.049936 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.049938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.050349 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.050350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.050405 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.050836 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:29.050841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:29.052002 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:29.054484 (+ 2482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.054485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.054485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.055182 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.055394 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.055394 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.055395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.055448 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.055753 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:29.055754 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:29.055794 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 04:05:29.055858 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 04:05:29.055939 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260927 04:05:29.056355 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.048352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.17.184.253:37421)
0927 04:05:29.048394 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.048396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.048404 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:05:29.048441 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.048442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.048451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.048497 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.048748 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.049320 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.050026 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.050233 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.050265 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.052342 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.052344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.052344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.052352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.054393 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.055261 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.055262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.055262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.055324 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.055513 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.055514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.055514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.055660 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.055726 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.056217 (+ 491us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.056295 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:05:29.056643 31520 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.056809 31520 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:29.056808 31670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:05:29.056903 31520 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8547b9858534eb8b435bafbfdab07c9, State: Initialized, Role: FOLLOWER
I20260927 04:05:29.057051 31520 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.057405 31520 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:29.057624 31677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.057783 31677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:05:29.059289 31688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:05:29.059386 31688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 04:05:29 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:05:29.060034 31452 master_runner.cc:186] Error getting master registration for 127.17.184.253:37421: OK, instance_id { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" instance_seqno: 1790481928864662 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:05:29.062793 31452 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 04:05:29 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:29.064853 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.057095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.17.184.252:46227)
0927 04:05:29.057125 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.057127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.057159 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:05:29.057216 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.057217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.057225 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.057285 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.057522 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.058220 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.059149 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.059428 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.059489 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.061654 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.061655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.061655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.061661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.063360 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.063731 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.063732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.063733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.063793 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.064190 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.064191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.064191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.064376 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.064455 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.064630 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.064749 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":15}
I20260927 04:05:29.064747 31672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.057070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56726)
0927 04:05:29.057119 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:29.057128 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:29.057171 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:29.057301 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:29.057301 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:29.057427 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:29.057451 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.057452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.058326 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.058327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.059051 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.059053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.059530 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.059530 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.059579 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.059993 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:29.059998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:29.061560 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:29.063447 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.063447 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.063448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.063651 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.064033 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.064034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.064035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.064100 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.064459 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:29.064460 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:29.064510 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 04:05:29.064585 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 04:05:29.064683 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:05:29.066449 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.056162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.17.184.254:36321)
0927 04:05:29.056198 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.056200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.056424 (+ 224us) server_negotiation.cc:407] Connection header received
0927 04:05:29.056899 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.056900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.056911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.056942 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.057840 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.058589 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.059590 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.060186 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.060254 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.063490 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.063491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.063492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.063498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.065282 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.065660 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.065661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.065662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.065728 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.065951 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.065951 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.065952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.066062 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.066228 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.066267 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.066344 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:05:29.066771 31670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9: Bootstrap starting.
I20260927 04:05:29.067538 31670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:29.067854 31670 log.cc:824] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:29.068679 31670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9: No bootstrap required, opened a new log
I20260927 04:05:29.070763 31670 raft_consensus.cc:348] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.071010 31670 raft_consensus.cc:374] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:29.071125 31670 raft_consensus.cc:729] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b279712e1dfc42f29a6deb6961cb31a9, State: Initialized, Role: FOLLOWER
Sep 27 04:05:29 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:05:29.071310 31670 consensus_queue.cc:260] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.071666 31670 sys_catalog.cc:564] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:29.071655 31691 sys_catalog.cc:455] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.071874 31691 sys_catalog.cc:458] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:05:29.073581 31702 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:05:29.073695 31702 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:05:29.075208 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.067182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.17.184.253:37421)
0927 04:05:29.067243 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.067245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.067339 (+ 94us) server_negotiation.cc:407] Connection header received
0927 04:05:29.067403 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.067404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.067414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.067468 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.068017 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.068604 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.069309 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.069516 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.069601 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.071814 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.071815 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.071816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.071823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.073543 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.074066 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.074067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.074068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.074127 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.074340 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.074341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.074341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.074478 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.074647 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.075031 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.075105 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:05:29.075519 31602 master_runner.cc:186] Error getting master registration for 127.17.184.253:37421: OK, instance_id { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" instance_seqno: 1790481928864662 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:05:29.077111 31602 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:05:29.077929 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:05:29.091992 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.079433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.17.184.253:37421)
0927 04:05:29.079478 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.079480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.083458 (+ 3978us) server_negotiation.cc:407] Connection header received
0927 04:05:29.083609 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.083610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.083623 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.083672 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.084356 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.085225 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.086401 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.086717 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.086763 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.088602 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.088604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.088604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.088612 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.090480 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.090976 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.090978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.090979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.091041 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.091306 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.091307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.091308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.091469 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.091571 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.091811 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.091894 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:05:29.094046 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.083233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60806 (local address 127.17.184.254:36321)
0927 04:05:29.083280 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.083281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.084481 (+ 1200us) server_negotiation.cc:407] Connection header received
0927 04:05:29.084595 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.084596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.084608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.084640 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.085373 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.086169 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.087662 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.088028 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.088077 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.090333 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.090335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.090335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.090344 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.092382 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.093154 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.093155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.093155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.093212 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.093417 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.093417 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.093418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.093548 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.093635 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.093860 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.093939 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:05:29.098706 31595 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } attempt: 3
I20260927 04:05:29.099073 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.083490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.17.184.252:46227)
0927 04:05:29.083540 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.083542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.087224 (+ 3682us) server_negotiation.cc:407] Connection header received
0927 04:05:29.087373 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.087374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.087384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.087418 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.092512 (+ 5094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.093340 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.094418 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.094736 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.094782 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.096221 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.096223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.096223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.096231 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.097725 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.098111 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.098112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.098112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.098175 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.098396 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.098396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.098397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.098512 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.098636 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.098824 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.098962 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 04:05:29 dist-test-slave-zqrn krb5kdc[31439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481928, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:05:29.106350 31675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.099284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.17.184.252:46227)
0927 04:05:29.099574 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:29.099576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:29.099584 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:05:29.099601 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:29.099602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:29.099610 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:29.099689 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:29.100006 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.100562 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.101325 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.101540 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.101631 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.103560 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:29.103562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:29.103562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:29.103569 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:29.105125 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.105536 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.105537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.105537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.105600 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:29.105805 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:29.105806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:29.105806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:29.105925 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:29.106007 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:29.106169 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 04:05:29.106248 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260927 04:05:29.106352 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:29.099015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:46227 (local address 127.0.0.1:56750)
0927 04:05:29.099152 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:29.099165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:29.099209 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:29.099695 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:29.099697 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:29.099873 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:29.099919 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.099921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.100677 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.100678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.101210 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:29.101211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:29.101632 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:29.101634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:29.101701 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:29.102127 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:29.102132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:29.103445 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:29.105216 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.105217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.105218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.105447 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.105662 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:29.105662 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:29.105663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:29.105729 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:29.105999 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:29.106000 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:29.106055 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 04:05:29.106143 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 04:05:29.106247 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 04:05:29.108343 31595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf: Bootstrap starting.
I20260927 04:05:29.109061 31595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:29.109391 31595 log.cc:824] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:29.110291 31595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf: No bootstrap required, opened a new log
I20260927 04:05:29.111995 31595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.112140 31595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:29.112210 31595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539b934ea8024dd5bcab30a8ba987fbf, State: Initialized, Role: FOLLOWER
I20260927 04:05:29.112466 31595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.112816 31595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [sys.catalog]: configured and running, proceeding with master startup.
W20260927 04:05:29.114753 31723 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:05:29.114857 31723 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:05:29.117030 31527 master_runner.cc:263] No leader master found from master 539b934ea8024dd5bcab30a8ba987fbf
I20260927 04:05:29.117110 31527 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:05:29.125808 31712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.125945 31712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:05:29.196295 31473 tablet.cc:2369] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:05:29.209491 31677 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:29.209612 31677 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.209836 31677 leader_election.cc:288] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 539b934ea8024dd5bcab30a8ba987fbf (127.17.184.253:37421), b279712e1dfc42f29a6deb6961cb31a9 (127.17.184.252:46227)
I20260927 04:05:29.210417 31569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8547b9858534eb8b435bafbfdab07c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539b934ea8024dd5bcab30a8ba987fbf" is_pre_election: true
I20260927 04:05:29.210636 31569 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8547b9858534eb8b435bafbfdab07c9 in term 0.
I20260927 04:05:29.211050 31470 leader_election.cc:302] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 539b934ea8024dd5bcab30a8ba987fbf, d8547b9858534eb8b435bafbfdab07c9; no voters:
I20260927 04:05:29.211416 31644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8547b9858534eb8b435bafbfdab07c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b279712e1dfc42f29a6deb6961cb31a9" is_pre_election: true
I20260927 04:05:29.211556 31644 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8547b9858534eb8b435bafbfdab07c9 in term 0.
I20260927 04:05:29.211834 31677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:05:29.211895 31677 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:05:29.211918 31677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:29.213066 31677 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.213219 31677 leader_election.cc:288] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [CANDIDATE]: Term 1 election: Requested vote from peers 539b934ea8024dd5bcab30a8ba987fbf (127.17.184.253:37421), b279712e1dfc42f29a6deb6961cb31a9 (127.17.184.252:46227)
I20260927 04:05:29.213608 31569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8547b9858534eb8b435bafbfdab07c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "539b934ea8024dd5bcab30a8ba987fbf"
I20260927 04:05:29.213687 31569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:29.214617 31569 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8547b9858534eb8b435bafbfdab07c9 in term 1.
I20260927 04:05:29.214843 31470 leader_election.cc:302] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 539b934ea8024dd5bcab30a8ba987fbf, d8547b9858534eb8b435bafbfdab07c9; no voters:
I20260927 04:05:29.215111 31644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d8547b9858534eb8b435bafbfdab07c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b279712e1dfc42f29a6deb6961cb31a9"
I20260927 04:05:29.215207 31644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:29.216048 31644 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8547b9858534eb8b435bafbfdab07c9 in term 1.
I20260927 04:05:29.216352 31677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:29.216485 31677 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [term 1 LEADER]: Becoming Leader. State: Replica: d8547b9858534eb8b435bafbfdab07c9, State: Running, Role: LEADER
I20260927 04:05:29.216593 31677 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } }
I20260927 04:05:29.216965 31677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8547b9858534eb8b435bafbfdab07c9. Latest consensus state: current_term: 1 leader_uuid: "d8547b9858534eb8b435bafbfdab07c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.217031 31677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:29.218396 31727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:05:29.218664 31727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:05:29.220922 31644 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [term 1 FOLLOWER]: Refusing update from remote peer d8547b9858534eb8b435bafbfdab07c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:29.221318 31677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:29.222788 31691 sys_catalog.cc:455] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8547b9858534eb8b435bafbfdab07c9. Latest consensus state: current_term: 1 leader_uuid: "d8547b9858534eb8b435bafbfdab07c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.222870 31691 sys_catalog.cc:458] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:05:29.223348 31569 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [term 1 FOLLOWER]: Refusing update from remote peer d8547b9858534eb8b435bafbfdab07c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:29.223624 31677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:29.224838 31712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8547b9858534eb8b435bafbfdab07c9. Latest consensus state: current_term: 1 leader_uuid: "d8547b9858534eb8b435bafbfdab07c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.224928 31712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:05:29.226449 31725 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8547b9858534eb8b435bafbfdab07c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.226552 31725 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:29.227175 31712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8547b9858534eb8b435bafbfdab07c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.227270 31712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:05:29.227551 31728 mvcc.cc:205] Tried to move back new op lower bound from 7333813982086696960 to 7333813982071783424. Current Snapshot: MvccSnapshot[applied={T|T < 7333813982086696960}]
I20260927 04:05:29.227711 31731 mvcc.cc:205] Tried to move back new op lower bound from 7333813982086696960 to 7333813982071783424. Current Snapshot: MvccSnapshot[applied={T|T < 7333813982086696960}]
I20260927 04:05:29.228787 31727 catalog_manager.cc:1347] Generated new cluster ID: 15f626103dc5476399b692825e07c743
I20260927 04:05:29.228837 31727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:05:29.231670 31677 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d8547b9858534eb8b435bafbfdab07c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.231752 31677 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:29.232414 31691 sys_catalog.cc:455] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d8547b9858534eb8b435bafbfdab07c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8547b9858534eb8b435bafbfdab07c9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 36321 } } peers { permanent_uuid: "539b934ea8024dd5bcab30a8ba987fbf" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 37421 } } peers { permanent_uuid: "b279712e1dfc42f29a6deb6961cb31a9" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 46227 } } }
I20260927 04:05:29.232517 31691 sys_catalog.cc:458] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:05:29.288959 31727 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:05:29.289625 31727 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:05:29.296921 31727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8547b9858534eb8b435bafbfdab07c9: Generated new TSK 0
I20260927 04:05:29.297161 31727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:05:30.074159 31702 catalog_manager.cc:1259] Loaded cluster ID: 15f626103dc5476399b692825e07c743
I20260927 04:05:30.074234 31702 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9: loading cluster ID for follower catalog manager: success
I20260927 04:05:30.075016 31702 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9: acquiring CA information for follower catalog manager: success
I20260927 04:05:30.075295 31702 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b279712e1dfc42f29a6deb6961cb31a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:05:30.115355 31723 catalog_manager.cc:1259] Loaded cluster ID: 15f626103dc5476399b692825e07c743
I20260927 04:05:30.115432 31723 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf: loading cluster ID for follower catalog manager: success
I20260927 04:05:30.116186 31723 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf: acquiring CA information for follower catalog manager: success
I20260927 04:05:30.116437 31723 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 539b934ea8024dd5bcab30a8ba987fbf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:05:30.360687 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 31452
I20260927 04:05:30.366040 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 31527
I20260927 04:05:30.370782 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 31602
2026-09-27T04:05:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2981 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:05:30 dist-test-slave-zqrn krb5kdc[31742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:05:30 dist-test-slave-zqrn krb5kdc[31742](info): set up 2 sockets
Sep 27 04:05:30 dist-test-slave-zqrn krb5kdc[31742](info): commencing operation
krb5kdc: starting...
W20260927 04:05:31.459053 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:05:31 dist-test-slave-zqrn krb5kdc[31742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:05:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:05:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:05:31.578135 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:40233
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:31.660066 31755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:31.660302 31755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:31.660342 31755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:31.661526 31755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:31.661597 31755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:31.661624 31755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:31.661654 31755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:31.661679 31755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:31.663261 31755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:31.663537 31755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:31.663770 31755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:31.668980 31761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:31.669021 31763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:31.668980 31760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:31.669209 31755 server_base.cc:1034] running on GCE node
I20260927 04:05:31.669605 31755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:31.669912 31755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:31.671056 31755 hybrid_clock.cc:648] HybridClock initialized: now 1790481931671034 us; error 39 us; skew 500 ppm
Sep 27 04:05:31 dist-test-slave-zqrn krb5kdc[31742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481931, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:31.673782 31755 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:05:31.674265 31755 webserver.cc:460] Webserver started at http://127.17.184.254:42937/ using document root <none> and password file <none>
I20260927 04:05:31.674499 31755 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:31.674576 31755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:31.674733 31755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:31.675885 31755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "139eb8e5d8dc45c28282b15a5914c8af"
format_stamp: "Formatted at 2026-09-27 04:05:31 on dist-test-slave-zqrn"
server_key: "8ce6a3e8aced8d1fb929a75bbb2eb0cc"
server_key_iv: "e8e285bc009252c47bd8540ad300555f"
server_key_version: "encryptionkey@0"
I20260927 04:05:31.676256 31755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "139eb8e5d8dc45c28282b15a5914c8af"
format_stamp: "Formatted at 2026-09-27 04:05:31 on dist-test-slave-zqrn"
server_key: "8ce6a3e8aced8d1fb929a75bbb2eb0cc"
server_key_iv: "e8e285bc009252c47bd8540ad300555f"
server_key_version: "encryptionkey@0"
I20260927 04:05:31.677693 31755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:05:31.678500 31770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:31.678762 31755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:31.678876 31755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "139eb8e5d8dc45c28282b15a5914c8af"
format_stamp: "Formatted at 2026-09-27 04:05:31 on dist-test-slave-zqrn"
server_key: "8ce6a3e8aced8d1fb929a75bbb2eb0cc"
server_key_iv: "e8e285bc009252c47bd8540ad300555f"
server_key_version: "encryptionkey@0"
I20260927 04:05:31.678985 31755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:31.703446 31755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:31.705673 31755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:31.705842 31755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:31.710104 31755 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:40233
I20260927 04:05:31.710144 31822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:40233 every 8 connection(s)
I20260927 04:05:31.710493 31755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:05:31.711426 31823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:31.713101 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 31755
I20260927 04:05:31.713215 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:05:31.713407 18147 external_mini_cluster.cc:1428] Setting key a6cc89c286c7a73593038d7191049ae6
I20260927 04:05:31.714279 31823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af: Bootstrap starting.
I20260927 04:05:31.714833 31823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:31.715152 31823 log.cc:824] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:31.715926 31823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af: No bootstrap required, opened a new log
I20260927 04:05:31.718190 31823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139eb8e5d8dc45c28282b15a5914c8af" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40233 } }
I20260927 04:05:31.718356 31823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:31.718394 31823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139eb8e5d8dc45c28282b15a5914c8af, State: Initialized, Role: FOLLOWER
I20260927 04:05:31.718559 31823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139eb8e5d8dc45c28282b15a5914c8af" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40233 } }
I20260927 04:05:31.718681 31823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:05:31.718727 31823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:05:31.718852 31823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:31.719455 31823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139eb8e5d8dc45c28282b15a5914c8af" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40233 } }
I20260927 04:05:31.719592 31823 leader_election.cc:302] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 139eb8e5d8dc45c28282b15a5914c8af; no voters:
I20260927 04:05:31.719753 31823 leader_election.cc:288] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:05:31.719817 31828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:31.719985 31828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [term 1 LEADER]: Becoming Leader. State: Replica: 139eb8e5d8dc45c28282b15a5914c8af, State: Running, Role: LEADER
I20260927 04:05:31.720041 31823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:31.720222 31828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139eb8e5d8dc45c28282b15a5914c8af" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40233 } }
Sep 27 04:05:31 dist-test-slave-zqrn krb5kdc[31742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:31.721000 31830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 139eb8e5d8dc45c28282b15a5914c8af. Latest consensus state: current_term: 1 leader_uuid: "139eb8e5d8dc45c28282b15a5914c8af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139eb8e5d8dc45c28282b15a5914c8af" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40233 } } }
I20260927 04:05:31.721081 31830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:31.721379 31838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:05:31.722038 31838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:05:31.721807 31829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "139eb8e5d8dc45c28282b15a5914c8af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "139eb8e5d8dc45c28282b15a5914c8af" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40233 } } }
I20260927 04:05:31.722224 31829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:31.723760 31838 catalog_manager.cc:1347] Generated new cluster ID: 851ec3928e9c43d08e006ac7a169c733
I20260927 04:05:31.723811 31838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:05:31.725696 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:31.713962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.17.184.254:40233)
0927 04:05:31.714161 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:31.714169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:31.714192 (+ 23us) server_negotiation.cc:407] Connection header received
0927 04:05:31.714645 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:31.714651 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:31.714700 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:31.714845 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:31.715655 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:31.717011 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:31.717771 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:31.718402 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:31.718509 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:31.721549 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:31.721553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:31.721553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:31.721566 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:31.724033 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:31.724452 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:31.724455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:31.724455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:31.724563 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:31.724780 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:31.724781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:31.724781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:31.725062 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:31.725211 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:31.725344 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 04:05:31.725472 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":84,"threads_started":1}
I20260927 04:05:31.726763 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:05:31.729977 31838 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:05:31.730499 31838 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:05:31.736027 31838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 139eb8e5d8dc45c28282b15a5914c8af: Generated new TSK 0
I20260927 04:05:31.736227 31838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:05:31.780963 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 31755
2026-09-27T04:05:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1408 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:05:31 dist-test-slave-zqrn krb5kdc[31850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:05:31 dist-test-slave-zqrn krb5kdc[31850](info): set up 2 sockets
Sep 27 04:05:31 dist-test-slave-zqrn krb5kdc[31850](info): commencing operation
krb5kdc: starting...
W20260927 04:05:32.855013 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:05:32 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:05:32.979040 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41965
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:33.060618 31863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:33.060837 31863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:33.060860 31863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:33.062057 31863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:33.062110 31863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:33.062126 31863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:33.062139 31863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:33.062152 31863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:33.063767 31863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:33.064035 31863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:33.064284 31863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:33.069375 31868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:33.069327 31871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:33.069327 31869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:33.069686 31863 server_base.cc:1034] running on GCE node
I20260927 04:05:33.069938 31863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:33.070233 31863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:33.071408 31863 hybrid_clock.cc:648] HybridClock initialized: now 1790481933071407 us; error 46 us; skew 500 ppm
Sep 27 04:05:33 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481933, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:33.074270 31863 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:05:33.074782 31863 webserver.cc:460] Webserver started at http://127.17.184.254:37785/ using document root <none> and password file <none>
I20260927 04:05:33.075068 31863 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:33.075136 31863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:33.075274 31863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:33.076725 31863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "365d53ef8dc548d0a6ec08f277478b10"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "a66c0c86a0ad6b65ee92981755dd1e68"
server_key_iv: "24915451f998833ab1cfd31d2c78aa2e"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.077203 31863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "365d53ef8dc548d0a6ec08f277478b10"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "a66c0c86a0ad6b65ee92981755dd1e68"
server_key_iv: "24915451f998833ab1cfd31d2c78aa2e"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.078779 31863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260927 04:05:33.079763 31878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.079993 31863 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:05:33.080121 31863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "365d53ef8dc548d0a6ec08f277478b10"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "a66c0c86a0ad6b65ee92981755dd1e68"
server_key_iv: "24915451f998833ab1cfd31d2c78aa2e"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.080204 31863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:33.106269 31863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:33.108215 31863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:33.108384 31863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:33.112584 31863 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:41965
I20260927 04:05:33.112637 31930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:41965 every 8 connection(s)
I20260927 04:05:33.113018 31863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:05:33.114046 31931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:33.114234 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 31863
I20260927 04:05:33.114354 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:05:33.114544 18147 external_mini_cluster.cc:1428] Setting key 8c4626ac8a87414fc4b8b23d7ff73442
I20260927 04:05:33.117740 31931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Bootstrap starting.
I20260927 04:05:33.118525 31931 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:33.118880 31931 log.cc:824] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:33.119760 31931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: No bootstrap required, opened a new log
Sep 27 04:05:33 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:33.121573 31931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:33.121752 31931 raft_consensus.cc:374] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:33.121797 31931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365d53ef8dc548d0a6ec08f277478b10, State: Initialized, Role: FOLLOWER
I20260927 04:05:33.121920 31931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:33.122026 31931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:05:33.122098 31931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:05:33.122154 31931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:33.122836 31931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:33.122974 31931 leader_election.cc:302] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 365d53ef8dc548d0a6ec08f277478b10; no voters:
I20260927 04:05:33.123153 31931 leader_election.cc:288] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:05:33.123243 31936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:33.123436 31931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:33.124619 31936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 1 LEADER]: Becoming Leader. State: Replica: 365d53ef8dc548d0a6ec08f277478b10, State: Running, Role: LEADER
I20260927 04:05:33.124729 31936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:33.125391 31937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "365d53ef8dc548d0a6ec08f277478b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } } }
I20260927 04:05:33.126150 31937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:33.125558 31948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365d53ef8dc548d0a6ec08f277478b10. Latest consensus state: current_term: 1 leader_uuid: "365d53ef8dc548d0a6ec08f277478b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } } }
W20260927 04:05:33.125756 31950 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 04:05:33.126441 31948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:33.126360 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.115084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.17.184.254:41965)
0927 04:05:33.115253 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:33.115268 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:33.115298 (+ 30us) server_negotiation.cc:407] Connection header received
0927 04:05:33.115739 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:33.115746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:33.115792 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:33.115908 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:33.116813 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.117657 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.118855 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.119345 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.119455 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.122135 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:33.122139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:33.122139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:33.122178 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:33.124458 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.124876 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.124879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.124879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.124979 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.125229 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.125230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.125231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.125539 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:33.125679 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:33.125950 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 04:05:33.126061 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
W20260927 04:05:33.126736 31950 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:05:33.126833 31953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:05:33.127017 31953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:05:33.128638 31953 catalog_manager.cc:1347] Generated new cluster ID: ca1be0ff67034f1482a26e3026c4f4f9
I20260927 04:05:33.128690 31953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:05:33.141383 31953 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:05:33.141947 31953 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:05:33.148833 31953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Generated new TSK 0
I20260927 04:05:33.149040 31953 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:05:33.178548 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41965
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:33.273232 31957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:33.273488 31957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:33.273531 31957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:33.274781 31957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:33.274853 31957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:33.274900 31957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:05:33.276518 31957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:33.276793 31957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:33.277058 31957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:33.283564 31963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:33.283530 31962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:33.283798 31965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:33.283779 31957 server_base.cc:1034] running on GCE node
I20260927 04:05:33.284044 31957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:33.284297 31957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:33.285467 31957 hybrid_clock.cc:648] HybridClock initialized: now 1790481933285458 us; error 44 us; skew 500 ppm
Sep 27 04:05:33 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481933, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:33.288240 31957 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:05:33.288751 31957 webserver.cc:460] Webserver started at http://127.17.184.193:40637/ using document root <none> and password file <none>
I20260927 04:05:33.289004 31957 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:33.289074 31957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:33.289247 31957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:33.290390 31957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "d7c742244c834c469d48bcf7a71f5709"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "6018527564250021b5d017e42b5da7a3"
server_key_iv: "be688a31aac4b2c5b1d608873df468fd"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.290756 31957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "d7c742244c834c469d48bcf7a71f5709"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "6018527564250021b5d017e42b5da7a3"
server_key_iv: "be688a31aac4b2c5b1d608873df468fd"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.292321 31957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:05:33.293156 31972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.293407 31957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:05:33.293510 31957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "d7c742244c834c469d48bcf7a71f5709"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "6018527564250021b5d017e42b5da7a3"
server_key_iv: "be688a31aac4b2c5b1d608873df468fd"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.293597 31957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:33.304400 31957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:33.306342 31957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:33.306505 31957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:33.306737 31957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:33.307163 31957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:33.307214 31957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.307255 31957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:33.307281 31957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.313092 31957 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:33715
I20260927 04:05:33.313141 32085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:33715 every 8 connection(s)
I20260927 04:05:33.313457 31957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:05:33.314308 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 31957
I20260927 04:05:33.314431 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:05:33.314637 18147 external_mini_cluster.cc:1428] Setting key 4a32785f4e0f2a0b9ffa3dce01778d89
Sep 27 04:05:33 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481933, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
I20260927 04:05:33.324424 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.314176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:44659 (local address 127.17.184.254:41965)
0927 04:05:33.314252 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:33.314255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:33.314458 (+ 203us) server_negotiation.cc:407] Connection header received
0927 04:05:33.315163 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:33.315164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:33.315175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:33.315253 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:33.316645 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.317470 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.318486 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.318720 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.318809 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.321093 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:33.321095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:33.321096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:33.321104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:33.322590 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.323192 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.323193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.323193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.323261 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.323661 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.323663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.323663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.323911 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:33.323986 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:33.324276 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 04:05:33.324350 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 04:05:33.324645 32088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.314140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:44659)
0927 04:05:33.314333 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:33.314378 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:33.315060 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:33.315334 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:33.315339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:33.316256 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:33.316537 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.316544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.317568 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.317569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.318388 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.318390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.318821 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.318822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.318979 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.319590 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:33.319598 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:33.320995 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:33.322716 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:33.322723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:33.322725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:33.323073 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:33.323402 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:33.323403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:33.323403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:33.323571 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:33.323987 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:33.323993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:33.324123 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 04:05:33.324214 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 04:05:33.324413 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260927 04:05:33.325306 32086 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41965
I20260927 04:05:33.325428 32086 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:33.325636 32086 heartbeater.cc:511] Master 127.17.184.254:41965 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
I20260927 04:05:33.326323 31895 ts_manager.cc:194] Registered new tserver with Master: d7c742244c834c469d48bcf7a71f5709 (127.17.184.193:33715)
I20260927 04:05:33.327373 31895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:44659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:05:33.343220 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41965
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:33.425238 32091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:33.425493 32091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:33.425534 32091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:33.426692 32091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:33.426759 32091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:33.426805 32091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:05:33.428339 32091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:33.428601 32091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:33.428836 32091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:33.433983 32099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:33.433964 32097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:33.433964 32096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:33.434232 32091 server_base.cc:1034] running on GCE node
I20260927 04:05:33.434500 32091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:33.434778 32091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:33.435958 32091 hybrid_clock.cc:648] HybridClock initialized: now 1790481933435943 us; error 26 us; skew 500 ppm
Sep 27 04:05:33 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481933, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:33.438580 32091 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:05:33.439173 32091 webserver.cc:460] Webserver started at http://127.17.184.194:35113/ using document root <none> and password file <none>
I20260927 04:05:33.439435 32091 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:33.439510 32091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:33.439678 32091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:33.440850 32091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "f54b524dbd3a4253a8138f101a109334"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "0994a6af09b6a9869922f27c898588bc"
server_key_iv: "8a998b4fd6054b922c094bd9b45bcb1a"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.441241 32091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "f54b524dbd3a4253a8138f101a109334"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "0994a6af09b6a9869922f27c898588bc"
server_key_iv: "8a998b4fd6054b922c094bd9b45bcb1a"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.442687 32091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:05:33.443521 32106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.443727 32091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:05:33.443864 32091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "f54b524dbd3a4253a8138f101a109334"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "0994a6af09b6a9869922f27c898588bc"
server_key_iv: "8a998b4fd6054b922c094bd9b45bcb1a"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.443962 32091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:33.458948 32091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:33.461246 32091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:33.461407 32091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:33.461658 32091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:33.461969 32091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:33.462024 32091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.462067 32091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:33.462092 32091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.467926 32091 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33059
I20260927 04:05:33.467967 32219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33059 every 8 connection(s)
I20260927 04:05:33.468322 32091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:05:33 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481933, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:33.477701 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.469050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:33241 (local address 127.17.184.254:41965)
0927 04:05:33.469175 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:33.469177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:33.469348 (+ 171us) server_negotiation.cc:407] Connection header received
0927 04:05:33.469929 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:33.469931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:33.469944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:33.469996 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:33.471016 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.471512 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.472520 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.472724 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.472802 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.474882 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:33.474884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:33.474884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:33.474914 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:33.476318 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.476740 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.476741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.476742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.476801 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.477048 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.477049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.477049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.477169 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:33.477248 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:33.477512 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 04:05:33.477594 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 04:05:33.477881 32222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.468977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:33241)
0927 04:05:33.469213 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:33.469264 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:33.469797 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:33.470065 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:33.470069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:33.470688 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:33.470884 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.470889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.471611 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.471612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.472417 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.472419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.472807 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.472808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.472932 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.473456 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:33.473465 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:33.474785 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:33.476421 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:33.476428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:33.476430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:33.476656 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:33.476868 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:33.476868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:33.476869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:33.476980 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:33.477237 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:33.477243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:33.477383 (+ 140us) client_negotiation.cc:769] Sending connection context
0927 04:05:33.477492 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 04:05:33.477646 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260927 04:05:33.478407 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 32091
I20260927 04:05:33.478413 32220 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41965
I20260927 04:05:33.478504 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:05:33.478511 32220 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:33.478704 32220 heartbeater.cc:511] Master 127.17.184.254:41965 requested a full tablet report, sending...
I20260927 04:05:33.478745 18147 external_mini_cluster.cc:1428] Setting key 23be8c85239c83acb308d856a3afa296
I20260927 04:05:33.479193 31894 ts_manager.cc:194] Registered new tserver with Master: f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059)
I20260927 04:05:33.479597 31894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:33241
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:05:33.503541 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41965
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:33.584012 32225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:33.584220 32225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:33.584245 32225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:33.585413 32225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:33.585456 32225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:33.585482 32225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:05:33.586941 32225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:33.587178 32225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:33.587404 32225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:05:33.592442 32225 server_base.cc:1034] running on GCE node
W20260927 04:05:33.592427 32231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:33.592419 32233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:33.592427 32230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:33.592878 32225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:33.593083 32225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:33.594223 32225 hybrid_clock.cc:648] HybridClock initialized: now 1790481933594210 us; error 27 us; skew 500 ppm
Sep 27 04:05:33 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481933, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:33.596911 32225 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:05:33.597417 32225 webserver.cc:460] Webserver started at http://127.17.184.195:39613/ using document root <none> and password file <none>
I20260927 04:05:33.597647 32225 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:33.597718 32225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:33.597849 32225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:33.598999 32225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "25b1e9f344c9804e83c4e6a2f5058d27"
server_key_iv: "83f09d3a037c62cba51702ccd3732ac5"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.599370 32225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "25b1e9f344c9804e83c4e6a2f5058d27"
server_key_iv: "83f09d3a037c62cba51702ccd3732ac5"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.600808 32225 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:05:33.601576 32240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.601766 32225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:33.601873 32225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "25b1e9f344c9804e83c4e6a2f5058d27"
server_key_iv: "83f09d3a037c62cba51702ccd3732ac5"
server_key_version: "encryptionkey@0"
I20260927 04:05:33.601948 32225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:33.617786 32225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:33.620191 32225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:33.620376 32225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:33.620646 32225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:33.621004 32225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:33.621043 32225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.621094 32225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:33.621125 32225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:33.628065 32225 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:44873
I20260927 04:05:33.628163 32353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:44873 every 8 connection(s)
I20260927 04:05:33.628515 32225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:05:33 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481933, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:33.638298 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.629254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:51293 (local address 127.17.184.254:41965)
0927 04:05:33.629300 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:33.629302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:33.629497 (+ 195us) server_negotiation.cc:407] Connection header received
0927 04:05:33.630074 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:33.630075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:33.630085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:33.630156 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:33.631136 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.631679 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.632646 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.632871 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.632930 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.635213 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:33.635214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:33.635214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:33.635225 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:33.636745 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.637239 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.637240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.637241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.637304 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.637582 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.637583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.637584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.637709 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:33.637790 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:33.638109 (+ 319us) server_negotiation.cc:299] Negotiation successful
0927 04:05:33.638193 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:05:33.638513 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.629207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:51293)
0927 04:05:33.629368 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:33.629410 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:33.629986 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:33.630211 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:33.630214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:33.630801 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:33.631054 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.631058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.631760 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.631761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.632563 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.632566 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.632954 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.632954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.633055 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.633540 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:33.633547 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:33.635104 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:33.636881 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:33.636888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:33.636890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:33.637138 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:33.637377 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:33.637379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:33.637379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:33.637508 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:33.637795 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:33.637802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:33.637955 (+ 153us) client_negotiation.cc:769] Sending connection context
0927 04:05:33.638073 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 04:05:33.638255 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260927 04:05:33.638702 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 32225
I20260927 04:05:33.638787 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:05:33.638994 18147 external_mini_cluster.cc:1428] Setting key 0f9bc3d96ee3aa64a9eecc88df2fa70d
I20260927 04:05:33.639011 32354 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41965
I20260927 04:05:33.639134 32354 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:33.639335 32354 heartbeater.cc:511] Master 127.17.184.254:41965 requested a full tablet report, sending...
I20260927 04:05:33.639745 31895 ts_manager.cc:194] Registered new tserver with Master: a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873)
I20260927 04:05:33.640168 31895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:51293
I20260927 04:05:33.650753 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:05:33.658316 31934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.652203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44388 (local address 127.17.184.254:41965)
0927 04:05:33.652302 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:33.652305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:33.652416 (+ 111us) server_negotiation.cc:407] Connection header received
0927 04:05:33.652488 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:33.652489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:33.652499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:33.652556 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:33.653157 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.653650 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.654383 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.654589 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.654662 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.655914 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:33.655916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:33.655916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:33.655923 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:33.657067 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.657441 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.657442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.657443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.657521 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:33.657728 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:33.657729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:33.657729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:33.657881 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:33.657987 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:33.658128 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 04:05:33.658211 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 04:05:33.659879 31895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:05:33.661554 31895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:05:33.668015 32368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.664059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33059 (local address 127.0.0.1:36962)
0927 04:05:33.664354 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:33.664364 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:33.664407 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:33.664898 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:33.664899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:33.664901 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:33.664952 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.664953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.665797 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.665798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.666981 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.666983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.667798 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.667799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.667854 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.667866 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 04:05:33.667920 (+ 54us) client_negotiation.cc:240] Negotiation successful
0927 04:05:33.667957 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260927 04:05:33.668095 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.664188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36962 (local address 127.17.184.194:33059)
0927 04:05:33.664682 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:33.664686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:33.664695 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:05:33.664731 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:33.664733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:33.664749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:33.664818 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:33.665034 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.665701 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.667046 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.667716 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.667795 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.667849 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:33.667943 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 04:05:33.667983 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":102,"threads_started":1}
I20260927 04:05:33.668385 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.663977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.17.184.195:44873)
0927 04:05:33.664408 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:33.664412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:33.664420 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:05:33.664455 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:33.664457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:33.664472 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:33.664550 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:33.664750 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.665899 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.667298 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.668009 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.668076 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.668126 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:33.668257 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 04:05:33.668303 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":80,"threads_started":1}
I20260927 04:05:33.668342 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.663839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44873 (local address 127.0.0.1:58766)
0927 04:05:33.663932 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:33.663969 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:33.664037 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:33.664570 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:33.664573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:33.664575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:33.664662 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.664664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.666025 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.666026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.667211 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.667214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.668094 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.668095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.668177 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.668182 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:05:33.668241 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 04:05:33.668281 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":105,"threads_started":1}
I20260927 04:05:33.669272 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.663841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33715 (local address 127.0.0.1:47200)
0927 04:05:33.664188 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:33.664200 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:33.664251 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:33.664659 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:33.664660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:33.664662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:33.664703 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.664704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.666036 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.666036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.667212 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:33.667214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.669039 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.669040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:33.669106 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.669109 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:05:33.669166 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 04:05:33.669203 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":2,"thread_start_us":47,"threads_started":1}
I20260927 04:05:33.669291 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:33.664176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.17.184.193:33715)
0927 04:05:33.664375 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:33.664380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:33.664391 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:05:33.664437 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:33.664439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:33.664455 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:33.664552 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:33.664782 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.665866 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.668164 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:33.668937 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:33.669032 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:33.669082 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:33.669180 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 04:05:33.669217 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260927 04:05:33.669199 32288 tablet_service.cc:1467] Processing CreateTablet for tablet 7e291f2903794bef9575f579fc2106e0 (DEFAULT_TABLE table=test-table [id=6189246ee4cc4bd6a643e0ed85b43cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:33.669199 32154 tablet_service.cc:1467] Processing CreateTablet for tablet 7e291f2903794bef9575f579fc2106e0 (DEFAULT_TABLE table=test-table [id=6189246ee4cc4bd6a643e0ed85b43cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:33.669534 32288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e291f2903794bef9575f579fc2106e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:33.669534 32154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e291f2903794bef9575f579fc2106e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:33.672998 32371 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Bootstrap starting.
I20260927 04:05:33.673331 32020 tablet_service.cc:1467] Processing CreateTablet for tablet 7e291f2903794bef9575f579fc2106e0 (DEFAULT_TABLE table=test-table [id=6189246ee4cc4bd6a643e0ed85b43cc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:33.673651 32020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e291f2903794bef9575f579fc2106e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:33.673666 32371 tablet_bootstrap.cc:654] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:33.674247 32371 log.cc:824] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:33.674609 32372 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Bootstrap starting.
I20260927 04:05:33.675212 32372 tablet_bootstrap.cc:654] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:33.675668 32372 log.cc:824] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:33.676152 32371 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: No bootstrap required, opened a new log
I20260927 04:05:33.676224 32371 ts_tablet_manager.cc:1397] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:05:33.676420 32372 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: No bootstrap required, opened a new log
I20260927 04:05:33.676483 32372 ts_tablet_manager.cc:1397] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:05:33.676792 32375 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Bootstrap starting.
I20260927 04:05:33.677477 32375 tablet_bootstrap.cc:654] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:33.677932 32375 log.cc:824] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:33.678270 32371 raft_consensus.cc:348] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:33.678445 32371 raft_consensus.cc:374] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:33.678370 32372 raft_consensus.cc:348] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:33.678490 32372 raft_consensus.cc:374] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:33.678488 32371 raft_consensus.cc:729] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54b524dbd3a4253a8138f101a109334, State: Initialized, Role: FOLLOWER
I20260927 04:05:33.678558 32372 raft_consensus.cc:729] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f46f6bb1b245bd8ec0022b6a213ab8, State: Initialized, Role: FOLLOWER
I20260927 04:05:33.678663 32375 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: No bootstrap required, opened a new log
I20260927 04:05:33.678674 32372 consensus_queue.cc:260] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:33.678731 32375 ts_tablet_manager.cc:1397] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:05:33.678792 32371 consensus_queue.cc:260] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:33.679014 32372 ts_tablet_manager.cc:1428] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:05:33.679024 32354 heartbeater.cc:503] Master 127.17.184.254:41965 was elected leader, sending a full tablet report...
I20260927 04:05:33.679203 32371 ts_tablet_manager.cc:1428] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:05:33.679421 32220 heartbeater.cc:503] Master 127.17.184.254:41965 was elected leader, sending a full tablet report...
I20260927 04:05:33.680595 32375 raft_consensus.cc:348] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:33.680720 32375 raft_consensus.cc:374] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:33.680742 32375 raft_consensus.cc:729] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c742244c834c469d48bcf7a71f5709, State: Initialized, Role: FOLLOWER
I20260927 04:05:33.680872 32375 consensus_queue.cc:260] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:33.681161 32086 heartbeater.cc:503] Master 127.17.184.254:41965 was elected leader, sending a full tablet report...
I20260927 04:05:33.681169 32375 ts_tablet_manager.cc:1428] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260927 04:05:33.719002 32221 tablet.cc:2369] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:05:33.814218 32087 tablet.cc:2369] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:05:33.879124 32355 tablet.cc:2369] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:05:34.009008 32379 raft_consensus.cc:482] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:34.009137 32379 raft_consensus.cc:504] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:34.009554 32379 leader_election.cc:288] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873), f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059)
I20260927 04:05:34.013880 32367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.009865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:51445 (local address 127.17.184.195:44873)
0927 04:05:34.009937 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.009939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.010030 (+ 91us) server_negotiation.cc:407] Connection header received
0927 04:05:34.010203 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.010205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.010217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.010285 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:34.010503 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.011276 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.012927 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.013491 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.013570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.013578 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.013766 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.013805 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 04:05:34.013880 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.010065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:36197 (local address 127.17.184.194:33059)
0927 04:05:34.010107 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.010110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.010163 (+ 53us) server_negotiation.cc:407] Connection header received
0927 04:05:34.010220 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.010221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.010230 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.010286 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:34.010503 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.011281 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.012926 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.013491 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.013570 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.013578 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.013766 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.013805 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 04:05:34.013901 32381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.009993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33059 (local address 127.17.184.193:36197)
0927 04:05:34.010100 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.010110 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.010164 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.010306 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.010307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.010308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:34.010420 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.010422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.011377 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.011378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.012834 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.012835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.013590 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.013591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.013689 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.013692 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.013745 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.013785 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":110,"thread_start_us":55,"threads_started":1}
I20260927 04:05:34.013901 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.009762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44873 (local address 127.17.184.193:51445)
0927 04:05:34.009933 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.009948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.010051 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.010305 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.010306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.010308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:34.010419 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.010421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.011377 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.011378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.012826 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.012829 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.013588 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.013590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.013664 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.013669 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.013734 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.013793 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":93,"thread_start_us":85,"threads_started":1}
I20260927 04:05:34.014284 32174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e291f2903794bef9575f579fc2106e0" candidate_uuid: "d7c742244c834c469d48bcf7a71f5709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54b524dbd3a4253a8138f101a109334" is_pre_election: true
I20260927 04:05:34.014454 32174 raft_consensus.cc:2454] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c742244c834c469d48bcf7a71f5709 in term 0.
I20260927 04:05:34.014705 31974 leader_election.cc:302] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c742244c834c469d48bcf7a71f5709, f54b524dbd3a4253a8138f101a109334; no voters:
I20260927 04:05:34.014848 32379 raft_consensus.cc:2781] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:05:34.014930 32379 raft_consensus.cc:482] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:05:34.014864 32308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e291f2903794bef9575f579fc2106e0" candidate_uuid: "d7c742244c834c469d48bcf7a71f5709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" is_pre_election: true
I20260927 04:05:34.014968 32379 raft_consensus.cc:3037] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:34.015041 32308 raft_consensus.cc:2454] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7c742244c834c469d48bcf7a71f5709 in term 0.
I20260927 04:05:34.015798 32379 raft_consensus.cc:504] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:34.015929 32379 leader_election.cc:288] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 1 election: Requested vote from peers a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873), f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059)
I20260927 04:05:34.016089 32174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e291f2903794bef9575f579fc2106e0" candidate_uuid: "d7c742244c834c469d48bcf7a71f5709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54b524dbd3a4253a8138f101a109334"
I20260927 04:05:34.016088 32308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e291f2903794bef9575f579fc2106e0" candidate_uuid: "d7c742244c834c469d48bcf7a71f5709" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8"
I20260927 04:05:34.016185 32308 raft_consensus.cc:3037] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:34.016177 32174 raft_consensus.cc:3037] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:34.017076 32174 raft_consensus.cc:2454] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c742244c834c469d48bcf7a71f5709 in term 1.
I20260927 04:05:34.017076 32308 raft_consensus.cc:2454] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7c742244c834c469d48bcf7a71f5709 in term 1.
I20260927 04:05:34.017262 31974 leader_election.cc:302] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f46f6bb1b245bd8ec0022b6a213ab8, d7c742244c834c469d48bcf7a71f5709; no voters:
I20260927 04:05:34.017354 32379 raft_consensus.cc:2781] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:34.017526 32379 raft_consensus.cc:686] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 1 LEADER]: Becoming Leader. State: Replica: d7c742244c834c469d48bcf7a71f5709, State: Running, Role: LEADER
I20260927 04:05:34.017635 32379 consensus_queue.cc:237] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:34.018445 31894 catalog_manager.cc:5495] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7c742244c834c469d48bcf7a71f5709 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "d7c742244c834c469d48bcf7a71f5709" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } health_report { overall_health: UNKNOWN } } }
I20260927 04:05:34.110651 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.108211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.17.184.193:33715)
0927 04:05:34.108296 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.108297 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.108306 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:05:34.108344 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.108345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.108362 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.108426 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:05:34.108565 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.109041 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.109876 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.110084 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.110142 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.110342 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:05:34.110453 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:05:34.110527 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.110570 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.110590 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 04:05:34.112744 32174 raft_consensus.cc:1261] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 1 FOLLOWER]: Refusing update from remote peer d7c742244c834c469d48bcf7a71f5709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:34.112746 32308 raft_consensus.cc:1261] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 1 FOLLOWER]: Refusing update from remote peer d7c742244c834c469d48bcf7a71f5709: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:34.113111 32379 consensus_queue.cc:1035] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:34.113281 32382 consensus_queue.cc:1035] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:34.116237 32386 mvcc.cc:205] Tried to move back new op lower bound from 7333814002122776576 to 7333814001736871936. Current Snapshot: MvccSnapshot[applied={T|T < 7333814002122776576}]
I20260927 04:05:34.116595 32387 mvcc.cc:205] Tried to move back new op lower bound from 7333814002122776576 to 7333814001736871936. Current Snapshot: MvccSnapshot[applied={T|T < 7333814002122776576}]
I20260927 04:05:34.118251 31895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:05:34.118340 31895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:05:34.119205 31895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Sending DeleteTablet for 3 replicas of tablet 7e291f2903794bef9575f579fc2106e0
I20260927 04:05:34.119547 32020 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e291f2903794bef9575f579fc2106e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:34 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:47200
I20260927 04:05:34.119645 32154 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e291f2903794bef9575f579fc2106e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:34 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:36962
I20260927 04:05:34.119635 32288 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e291f2903794bef9575f579fc2106e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:34 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:58766
I20260927 04:05:34.119916 32397 tablet_replica.cc:330] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: stopping tablet replica
I20260927 04:05:34.119927 32395 tablet_replica.cc:330] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: stopping tablet replica
I20260927 04:05:34.119993 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 31957
I20260927 04:05:34.120059 32395 raft_consensus.cc:2229] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:05:34.120064 32397 raft_consensus.cc:2229] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:05:34.120174 32397 raft_consensus.cc:2258] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:34.120680 32397 ts_tablet_manager.cc:1905] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:34.121495 32396 tablet_replica.cc:330] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: stopping tablet replica
I20260927 04:05:34.121616 32396 raft_consensus.cc:2229] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:05:34.121734 32396 raft_consensus.cc:2258] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:34.122740 32397 ts_tablet_manager.cc:1918] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:05:34.122822 32397 log.cc:1196] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/7e291f2903794bef9575f579fc2106e0
I20260927 04:05:34.123106 32397 ts_tablet_manager.cc:1939] T 7e291f2903794bef9575f579fc2106e0 P a8f46f6bb1b245bd8ec0022b6a213ab8: Deleting consensus metadata
I20260927 04:05:34.123279 32396 ts_tablet_manager.cc:1905] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:34.123575 31880 catalog_manager.cc:4841] TS a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873): tablet 7e291f2903794bef9575f579fc2106e0 (table test-table [id=6189246ee4cc4bd6a643e0ed85b43cc2]) successfully deleted
I20260927 04:05:34.124846 32396 ts_tablet_manager.cc:1918] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:05:34.124938 32396 log.cc:1196] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/7e291f2903794bef9575f579fc2106e0
I20260927 04:05:34.125171 32396 ts_tablet_manager.cc:1939] T 7e291f2903794bef9575f579fc2106e0 P f54b524dbd3a4253a8138f101a109334: Deleting consensus metadata
I20260927 04:05:34.125567 31880 catalog_manager.cc:4841] TS f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059): tablet 7e291f2903794bef9575f579fc2106e0 (table test-table [id=6189246ee4cc4bd6a643e0ed85b43cc2]) successfully deleted
W20260927 04:05:34.126102 31882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:33715 (error 108)
I20260927 04:05:34.126418 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 32091
I20260927 04:05:34.126595 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.126475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33715 (local address 127.0.0.1:47218)
0927 04:05:34.126523 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.126558 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 04:05:34.127208 31882 catalog_manager.cc:4584] TS d7c742244c834c469d48bcf7a71f5709 (127.17.184.193:33715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e291f2903794bef9575f579fc2106e0: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33715: connect: Connection refused (error 111)
I20260927 04:05:34.131983 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 32225
I20260927 04:05:34.137501 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 31863
I20260927 04:05:34.143184 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41965
--webserver_interface=127.17.184.254
--webserver_port=37785
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:34.225980 32399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:34.226207 32399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:34.226229 32399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:34.227456 32399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:34.227500 32399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:34.227514 32399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:34.227527 32399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:34.227540 32399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:34.229106 32399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:34.229377 32399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:34.229668 32399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:34.238556 32408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:34.238627 32406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:34.238653 32405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:34.238783 32399 server_base.cc:1034] running on GCE node
I20260927 04:05:34.239135 32399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:34.239499 32399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:34.240674 32399 hybrid_clock.cc:648] HybridClock initialized: now 1790481934240650 us; error 57 us; skew 500 ppm
Sep 27 04:05:34 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481934, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:34.243417 32399 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:05:34.243935 32399 webserver.cc:460] Webserver started at http://127.17.184.254:37785/ using document root <none> and password file <none>
I20260927 04:05:34.244159 32399 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:34.244216 32399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:34.245635 32399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:34.246358 32415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.246608 32399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:34.246732 32399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "365d53ef8dc548d0a6ec08f277478b10"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "a66c0c86a0ad6b65ee92981755dd1e68"
server_key_iv: "24915451f998833ab1cfd31d2c78aa2e"
server_key_version: "encryptionkey@0"
I20260927 04:05:34.247246 32399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:34.259481 32399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:34.261806 32399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:34.261963 32399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:34.266110 32399 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:41965
I20260927 04:05:34.266141 32467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:41965 every 8 connection(s)
I20260927 04:05:34.266522 32399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:05:34.268210 32468 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:05:34.268280 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 32399
I20260927 04:05:34.268556 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:33715
--local_ip_for_outbound_sockets=127.17.184.193
--tserver_master_addrs=127.17.184.254:41965
--webserver_port=40637
--webserver_interface=127.17.184.193
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:34.269215 32468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Bootstrap starting.
I20260927 04:05:34.277505 32468 log.cc:824] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:34.279489 32468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:05:34.279793 32468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Bootstrap complete.
I20260927 04:05:34.282105 32468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:34.282361 32468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365d53ef8dc548d0a6ec08f277478b10, State: Initialized, Role: FOLLOWER
I20260927 04:05:34.282545 32468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:34.282621 32468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:05:34.282677 32468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:05:34.282727 32468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:05:34.283771 32468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:34.283891 32468 leader_election.cc:302] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 365d53ef8dc548d0a6ec08f277478b10; no voters:
I20260927 04:05:34.284087 32468 leader_election.cc:288] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 04:05:34.284122 32472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 04:05:34.284233 32472 raft_consensus.cc:686] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 2 LEADER]: Becoming Leader. State: Replica: 365d53ef8dc548d0a6ec08f277478b10, State: Running, Role: LEADER
I20260927 04:05:34.284328 32472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:34.284364 32468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:34.284598 32474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365d53ef8dc548d0a6ec08f277478b10. Latest consensus state: current_term: 2 leader_uuid: "365d53ef8dc548d0a6ec08f277478b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } } }
I20260927 04:05:34.284688 32474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:34.284862 32473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "365d53ef8dc548d0a6ec08f277478b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } } }
I20260927 04:05:34.284914 32473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:34.286062 32487 catalog_manager.cc:1259] Loaded cluster ID: ca1be0ff67034f1482a26e3026c4f4f9
I20260927 04:05:34.286140 32487 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: loading cluster ID for follower catalog manager: success
W20260927 04:05:34.286350 32487 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260927 04:05:34.286396 32487 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260927 04:05:34.286434 32477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:05:34.286746 32477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:05:34.286800 32477 catalog_manager.cc:1259] Loaded cluster ID: ca1be0ff67034f1482a26e3026c4f4f9
I20260927 04:05:34.286854 32477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:05:34.287034 32477 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 04:05:34.289686 32477 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 04:05:34.291910 32477 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:05:34.292085 32477 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Loaded TSK: 0
I20260927 04:05:34.292348 32477 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 04:05:34.351868 32470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:34.352100 32470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:34.352139 32470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:34.353392 32470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:34.353451 32470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:34.353500 32470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:05:34.355022 32470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:33715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=40637
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:34.355288 32470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:34.355516 32470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:34.360565 32498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:34.360576 32495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:34.360736 32496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:34.361305 32470 server_base.cc:1034] running on GCE node
I20260927 04:05:34.361547 32470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:34.361799 32470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:34.362967 32470 hybrid_clock.cc:648] HybridClock initialized: now 1790481934362945 us; error 39 us; skew 500 ppm
Sep 27 04:05:34 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481934, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:34.365701 32470 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:05:34.366262 32470 webserver.cc:460] Webserver started at http://127.17.184.193:40637/ using document root <none> and password file <none>
I20260927 04:05:34.366500 32470 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:34.366580 32470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:34.368013 32470 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:34.368732 32505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.368911 32470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:34.369022 32470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "d7c742244c834c469d48bcf7a71f5709"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "6018527564250021b5d017e42b5da7a3"
server_key_iv: "be688a31aac4b2c5b1d608873df468fd"
server_key_version: "encryptionkey@0"
I20260927 04:05:34.369375 32470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:34.376133 32470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:34.378264 32470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:34.378418 32470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:34.378669 32470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:34.379252 32512 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:05:34.381225 32470 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:05:34.381309 32470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:05:34.381357 32470 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:05:34.382006 32470 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:05:34.382071 32470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:05:34.382128 32512 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Bootstrap starting.
I20260927 04:05:34.389422 32470 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:33715
I20260927 04:05:34.389484 32619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:33715 every 8 connection(s)
I20260927 04:05:34.389964 32470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:05:34.393213 32512 log.cc:824] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:34.393785 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 32470
I20260927 04:05:34.394078 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:33059
--local_ip_for_outbound_sockets=127.17.184.194
--tserver_master_addrs=127.17.184.254:41965
--webserver_port=35113
--webserver_interface=127.17.184.194
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:34.396202 32512 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:05:34.396564 32512 tablet_bootstrap.cc:492] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Bootstrap complete.
I20260927 04:05:34.397533 32512 ts_tablet_manager.cc:1397] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
Sep 27 04:05:34 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481934, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:34.398953 32512 raft_consensus.cc:348] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:34.399356 32512 raft_consensus.cc:729] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c742244c834c469d48bcf7a71f5709, State: Initialized, Role: FOLLOWER
I20260927 04:05:34.399514 32512 consensus_queue.cc:260] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } } peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } }
I20260927 04:05:34.399837 32512 ts_tablet_manager.cc:1428] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260927 04:05:34.402786 32622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.390684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:40933)
0927 04:05:34.391014 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.391066 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.391507 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.392023 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.392027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.392347 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:34.392530 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.392535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.393929 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.393930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.395193 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.395196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.396063 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.396064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.396234 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.396873 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:34.396883 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:34.398727 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:34.401068 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:34.401075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:34.401076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:34.401310 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:34.401598 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:34.401599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:34.401599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:34.401720 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:34.402159 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:34.402165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:34.402302 (+ 137us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.402412 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.402590 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260927 04:05:34.402799 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.390816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:40933 (local address 127.17.184.254:41965)
0927 04:05:34.391671 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.391678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.391705 (+ 27us) server_negotiation.cc:407] Connection header received
0927 04:05:34.391805 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.391809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.391848 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.392001 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:34.392662 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.393796 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.395308 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.395925 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.396041 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.398888 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:34.398932 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:34.398933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:34.398957 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:34.400956 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:34.401409 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:34.401411 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:34.401412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:34.401512 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:34.401792 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:34.401793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:34.401794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:34.402078 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:34.402181 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.402463 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.402590 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":108,"threads_started":1}
I20260927 04:05:34.403486 32620 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41965
I20260927 04:05:34.403577 32620 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:34.403753 32620 heartbeater.cc:511] Master 127.17.184.254:41965 requested a full tablet report, sending...
I20260927 04:05:34.404440 32432 ts_manager.cc:194] Registered new tserver with Master: d7c742244c834c469d48bcf7a71f5709 (127.17.184.193:33715)
I20260927 04:05:34.405579 32432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:40933
I20260927 04:05:34.406402 32620 heartbeater.cc:503] Master 127.17.184.254:41965 was elected leader, sending a full tablet report...
I20260927 04:05:34.410084 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.405013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33715 (local address 127.0.0.1:60076)
0927 04:05:34.406982 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.407020 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.407073 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.407308 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.407310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.407313 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:34.407397 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.407398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.408230 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.408231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.409057 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.409059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.409728 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.409730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.409829 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.409841 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.409947 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.410004 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1941,"thread_start_us":112,"threads_started":1}
I20260927 04:05:34.410075 32628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.405270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.17.184.193:33715)
0927 04:05:34.407171 (+ 1901us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.407175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.407182 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:05:34.407207 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.407208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.407219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.407304 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:34.407480 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.408135 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.409171 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.409632 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.409720 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.409777 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.409966 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.410005 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1877,"thread_start_us":62,"threads_started":1}
I20260927 04:05:34.410758 32554 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e291f2903794bef9575f579fc2106e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:34 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:60076
I20260927 04:05:34.411316 32629 tablet_replica.cc:330] stopping tablet replica
I20260927 04:05:34.411442 32629 raft_consensus.cc:2229] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:05:34.411510 32629 raft_consensus.cc:2258] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:34.411883 32629 ts_tablet_manager.cc:1905] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:34.413964 32629 ts_tablet_manager.cc:1918] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:05:34.414127 32629 log.cc:1196] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/7e291f2903794bef9575f579fc2106e0
I20260927 04:05:34.414397 32629 ts_tablet_manager.cc:1939] T 7e291f2903794bef9575f579fc2106e0 P d7c742244c834c469d48bcf7a71f5709: Deleting consensus metadata
I20260927 04:05:34.414994 32419 catalog_manager.cc:4841] TS d7c742244c834c469d48bcf7a71f5709 (127.17.184.193:33715): tablet 7e291f2903794bef9575f579fc2106e0 (table test-table [id=6189246ee4cc4bd6a643e0ed85b43cc2]) successfully deleted
W20260927 04:05:34.489748 32625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:34.489944 32625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:34.489967 32625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:34.491039 32625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:34.491073 32625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:34.491099 32625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:05:34.492466 32625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:33059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=35113
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:34.492650 32625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:34.492822 32625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:34.497726 32635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:34.497735 32637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:34.497829 32634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:34.498041 32625 server_base.cc:1034] running on GCE node
I20260927 04:05:34.498235 32625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:34.498451 32625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:34.499660 32625 hybrid_clock.cc:648] HybridClock initialized: now 1790481934499642 us; error 28 us; skew 500 ppm
Sep 27 04:05:34 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481934, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:34.502154 32625 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:05:34.502589 32625 webserver.cc:460] Webserver started at http://127.17.184.194:35113/ using document root <none> and password file <none>
I20260927 04:05:34.502826 32625 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:34.502928 32625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:34.504284 32625 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:05:34.504890 32644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.505054 32625 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:05:34.505156 32625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "f54b524dbd3a4253a8138f101a109334"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "0994a6af09b6a9869922f27c898588bc"
server_key_iv: "8a998b4fd6054b922c094bd9b45bcb1a"
server_key_version: "encryptionkey@0"
I20260927 04:05:34.505503 32625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:34.520332 32625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:34.522660 32625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:34.522825 32625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:34.523087 32625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:34.523396 32625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:34.523450 32625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.523491 32625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:34.523520 32625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.528997 32625 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33059
I20260927 04:05:34.529091 32757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33059 every 8 connection(s)
I20260927 04:05:34.529421 32625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:05:34.533911 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 32625
I20260927 04:05:34.534204 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:44873
--local_ip_for_outbound_sockets=127.17.184.195
--tserver_master_addrs=127.17.184.254:41965
--webserver_port=39613
--webserver_interface=127.17.184.195
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:05:34 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481934, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:34.539569 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.530187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:41289 (local address 127.17.184.254:41965)
0927 04:05:34.530255 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.530257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.530520 (+ 263us) server_negotiation.cc:407] Connection header received
0927 04:05:34.531146 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.531147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.531164 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.531227 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:34.532253 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.532891 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.533905 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.534189 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.534278 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.536559 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:34.536560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:34.536560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:34.536568 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:34.537997 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:34.538486 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:34.538487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:34.538488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:34.538548 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:34.538865 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:34.538866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:34.538866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:34.539013 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:34.539094 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.539373 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.539467 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:05:34.539799 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.530191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:41289)
0927 04:05:34.530367 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.530428 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.531025 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.531297 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.531301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.531943 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:34.532141 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.532145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.532981 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.532982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.533797 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.533799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.534288 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.534289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.534417 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.535047 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:34.535056 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:34.536448 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:34.538142 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:34.538150 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:34.538152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:34.538389 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:34.538676 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:34.538677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:34.538678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:34.538793 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:34.539085 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:34.539091 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:34.539241 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.539352 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.539507 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260927 04:05:34.540333 32758 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41965
I20260927 04:05:34.540413 32758 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:34.540613 32758 heartbeater.cc:511] Master 127.17.184.254:41965 requested a full tablet report, sending...
I20260927 04:05:34.541036 32432 ts_manager.cc:194] Registered new tserver with Master: f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059)
I20260927 04:05:34.541599 32432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:41289
W20260927 04:05:34.616714 32761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:34.616925 32761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:34.616947 32761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:34.618016 32761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:34.618049 32761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:34.618077 32761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:05:34.619639 32761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:44873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=39613
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:34.619829 32761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:34.619997 32761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:34.624934 32766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:34.625072 301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:34.625023 32767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:34.625355 32761 server_base.cc:1034] running on GCE node
I20260927 04:05:34.625545 32761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:34.625805 32761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:34.626951 32761 hybrid_clock.cc:648] HybridClock initialized: now 1790481934626936 us; error 36 us; skew 500 ppm
Sep 27 04:05:34 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481934, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:34.629354 32761 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:05:34.629870 32761 webserver.cc:460] Webserver started at http://127.17.184.195:39613/ using document root <none> and password file <none>
I20260927 04:05:34.630108 32761 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:34.630189 32761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:34.631613 32761 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:05:34.632311 308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.632481 32761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:05:34.632584 32761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "25b1e9f344c9804e83c4e6a2f5058d27"
server_key_iv: "83f09d3a037c62cba51702ccd3732ac5"
server_key_version: "encryptionkey@0"
I20260927 04:05:34.632969 32761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:34.663573 32761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:34.665570 32761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:34.665740 32761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:34.665985 32761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:34.666292 32761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:34.666344 32761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.666379 32761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:34.666407 32761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.672473 32761 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:44873
I20260927 04:05:34.672559 421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:44873 every 8 connection(s)
I20260927 04:05:34.672865 32761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:05:34 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481934, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:34.680094 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 32761
I20260927 04:05:34.682228 32623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.673520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:58647 (local address 127.17.184.254:41965)
0927 04:05:34.673594 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.673596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.673831 (+ 235us) server_negotiation.cc:407] Connection header received
0927 04:05:34.674290 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.674291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.674301 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.674354 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:34.675345 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.675839 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.676866 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.677062 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.677137 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.679273 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:34.679274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:34.679275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:34.679283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:34.680801 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:34.681308 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:34.681309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:34.681309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:34.681359 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:34.681627 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:34.681627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:34.681628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:34.681744 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:34.681813 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.682051 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.682135 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 04:05:34.682334 424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.673523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:58647)
0927 04:05:34.673706 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.673741 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.674206 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.674420 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.674423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.675050 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:34.675253 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.675257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.675936 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.675938 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.676767 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.676769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.677151 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.677152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.677258 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.677715 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:34.677723 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:34.679177 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:34.680875 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:34.680882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:34.680884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:34.681227 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:34.681420 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:34.681421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:34.681421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:34.681563 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:34.681811 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:34.681816 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:34.681928 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.682010 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.682123 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":107,"threads_started":1}
I20260927 04:05:34.682775 422 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41965
I20260927 04:05:34.682889 422 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:34.683087 422 heartbeater.cc:511] Master 127.17.184.254:41965 requested a full tablet report, sending...
I20260927 04:05:34.683497 32432 ts_manager.cc:194] Registered new tserver with Master: a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873)
I20260927 04:05:34.683992 32432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:58647
I20260927 04:05:34.687160 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.680537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34578 (local address 127.17.184.254:41965)
0927 04:05:34.680649 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.680651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.680691 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:05:34.680746 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.680747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.680759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.680828 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:34.681532 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.682219 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.683348 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.683572 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.683664 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.684973 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:34.684974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:34.684974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:34.684980 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:34.686172 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:34.686506 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:34.686507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:34.686507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:34.686556 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:34.686724 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:34.686724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:34.686725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:34.686813 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:34.686980 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.687033 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.687096 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
I20260927 04:05:34.687930 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:05:34.690743 426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.688253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.17.184.254:41965)
0927 04:05:34.688327 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.688329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.688422 (+ 93us) server_negotiation.cc:407] Connection header received
0927 04:05:34.688493 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.688494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.688526 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.688607 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:05:34.688747 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.689199 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.689961 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.690173 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.690262 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.690431 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:05:34.690533 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:05:34.690626 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.690651 (+ 25us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.690672 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 04:05:34.691041 32432 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:34592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:05:34.692554 32432 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:05:34.694684 32554 tablet_service.cc:1467] Processing CreateTablet for tablet e82786dd934240a58a240cd000dcfd8f (DEFAULT_TABLE table=test-table [id=b2255942661c433283d8eefe2145a368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:34.694872 32554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82786dd934240a58a240cd000dcfd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:34.696221 32512 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Bootstrap starting.
I20260927 04:05:34.696532 32512 tablet_bootstrap.cc:654] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:34.697145 32512 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: No bootstrap required, opened a new log
I20260927 04:05:34.697196 32512 ts_tablet_manager.cc:1397] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 04:05:34.697290 32512 raft_consensus.cc:348] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.697362 32512 raft_consensus.cc:374] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:34.697413 32512 raft_consensus.cc:729] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c742244c834c469d48bcf7a71f5709, State: Initialized, Role: FOLLOWER
I20260927 04:05:34.697455 32512 consensus_queue.cc:260] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.697536 32512 ts_tablet_manager.cc:1428] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:34.698668 32627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.694305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33059 (local address 127.0.0.1:50606)
0927 04:05:34.694364 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.694376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.694412 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.694964 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.694965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.694966 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:34.695004 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.695006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.696440 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.696442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.697490 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.697493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.698415 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.698416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.698461 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.698464 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.698542 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.698580 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 04:05:34.698774 428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.694433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44873 (local address 127.0.0.1:48116)
0927 04:05:34.694594 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.694604 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.694652 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.695055 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.695056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.695058 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:34.695102 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.695104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.696415 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.696416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.697585 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.697588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.698547 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.698547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.698598 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.698600 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.698651 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.698685 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":46,"thread_start_us":93,"threads_started":1}
I20260927 04:05:34.698698 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.694608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50606 (local address 127.17.184.194:33059)
0927 04:05:34.694749 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.694755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.694767 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:05:34.694814 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.694817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.694835 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.694966 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:34.695070 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.696309 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.697590 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.698335 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.698415 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.698471 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.698559 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.698605 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260927 04:05:34.699117 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.694528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.17.184.195:44873)
0927 04:05:34.694794 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.694798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.694807 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:05:34.694840 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.694841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.694855 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.694988 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:34.695210 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.696309 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.697661 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.698469 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.698539 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.698586 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.698660 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.698701 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":79,"threads_started":1}
I20260927 04:05:34.699551 32692 tablet_service.cc:1467] Processing CreateTablet for tablet e82786dd934240a58a240cd000dcfd8f (DEFAULT_TABLE table=test-table [id=b2255942661c433283d8eefe2145a368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:34.699684 356 tablet_service.cc:1467] Processing CreateTablet for tablet e82786dd934240a58a240cd000dcfd8f (DEFAULT_TABLE table=test-table [id=b2255942661c433283d8eefe2145a368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:34.699910 32692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82786dd934240a58a240cd000dcfd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:34.699973 356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e82786dd934240a58a240cd000dcfd8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:34.703388 432 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Bootstrap starting.
I20260927 04:05:34.703475 433 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Bootstrap starting.
I20260927 04:05:34.704119 432 tablet_bootstrap.cc:654] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:34.704119 433 tablet_bootstrap.cc:654] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:34.704763 432 log.cc:824] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:34.704763 433 log.cc:824] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:34.705530 432 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: No bootstrap required, opened a new log
I20260927 04:05:34.705605 432 ts_tablet_manager.cc:1397] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:05:34.705675 433 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: No bootstrap required, opened a new log
I20260927 04:05:34.705732 433 ts_tablet_manager.cc:1397] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:05:34.707665 433 raft_consensus.cc:348] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.707665 432 raft_consensus.cc:348] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.707839 432 raft_consensus.cc:374] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:34.707839 433 raft_consensus.cc:374] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:34.707875 433 raft_consensus.cc:729] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54b524dbd3a4253a8138f101a109334, State: Initialized, Role: FOLLOWER
I20260927 04:05:34.707875 432 raft_consensus.cc:729] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8f46f6bb1b245bd8ec0022b6a213ab8, State: Initialized, Role: FOLLOWER
I20260927 04:05:34.708000 433 consensus_queue.cc:260] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.708000 432 consensus_queue.cc:260] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.708290 432 ts_tablet_manager.cc:1428] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:05:34.708290 433 ts_tablet_manager.cc:1428] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:05:34.708317 422 heartbeater.cc:503] Master 127.17.184.254:41965 was elected leader, sending a full tablet report...
I20260927 04:05:34.708403 32758 heartbeater.cc:503] Master 127.17.184.254:41965 was elected leader, sending a full tablet report...
W20260927 04:05:34.780035 32759 tablet.cc:2369] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:05:34.869719 437 raft_consensus.cc:482] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:34.869899 437 raft_consensus.cc:504] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.870361 437 leader_election.cc:288] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873), d7c742244c834c469d48bcf7a71f5709 (127.17.184.193:33715)
I20260927 04:05:34.873591 32628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.870653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:41403 (local address 127.17.184.193:33715)
0927 04:05:34.870700 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.870703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.870852 (+ 149us) server_negotiation.cc:407] Connection header received
0927 04:05:34.870928 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.870929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.870941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.871004 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:34.871204 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.871660 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.872750 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.873225 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.873315 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.873324 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.873483 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.873521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:05:34.873592 438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.870558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33715 (local address 127.17.184.194:41403)
0927 04:05:34.870760 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.870770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.870840 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.871017 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.871017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.871019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:34.871115 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.871117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.871754 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.871755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.872649 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.872652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.873316 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.873317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.873397 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.873400 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.873463 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.873503 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":42,"thread_start_us":107,"threads_started":1}
I20260927 04:05:34.873836 32760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.870545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44873 (local address 127.17.184.194:53979)
0927 04:05:34.870611 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.870624 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:34.870673 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:34.870869 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:34.870870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:34.870872 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:34.870992 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.871002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.871767 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.871768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.872741 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:34.872744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.873522 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.873523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:34.873612 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.873621 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 04:05:34.873698 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 04:05:34.873752 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":23}
I20260927 04:05:34.873927 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.870680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:53979 (local address 127.17.184.195:44873)
0927 04:05:34.870730 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.870732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.870743 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:05:34.870776 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.870778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.870790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.870851 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:34.871079 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.871657 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.872854 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.873394 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.873525 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.873534 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.873816 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.873859 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:05:34.873873 32574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82786dd934240a58a240cd000dcfd8f" candidate_uuid: "f54b524dbd3a4253a8138f101a109334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c742244c834c469d48bcf7a71f5709" is_pre_election: true
I20260927 04:05:34.874054 32574 raft_consensus.cc:2454] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54b524dbd3a4253a8138f101a109334 in term 0.
I20260927 04:05:34.874207 376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82786dd934240a58a240cd000dcfd8f" candidate_uuid: "f54b524dbd3a4253a8138f101a109334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" is_pre_election: true
I20260927 04:05:34.874277 32648 leader_election.cc:302] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7c742244c834c469d48bcf7a71f5709, f54b524dbd3a4253a8138f101a109334; no voters:
I20260927 04:05:34.874385 376 raft_consensus.cc:2454] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f54b524dbd3a4253a8138f101a109334 in term 0.
I20260927 04:05:34.874420 437 raft_consensus.cc:2781] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:05:34.874475 437 raft_consensus.cc:482] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:05:34.874501 437 raft_consensus.cc:3037] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:34.875300 437 raft_consensus.cc:504] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.875519 437 leader_election.cc:288] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [CANDIDATE]: Term 1 election: Requested vote from peers a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873), d7c742244c834c469d48bcf7a71f5709 (127.17.184.193:33715)
I20260927 04:05:34.875699 32574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82786dd934240a58a240cd000dcfd8f" candidate_uuid: "f54b524dbd3a4253a8138f101a109334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c742244c834c469d48bcf7a71f5709"
I20260927 04:05:34.875700 376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e82786dd934240a58a240cd000dcfd8f" candidate_uuid: "f54b524dbd3a4253a8138f101a109334" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8"
I20260927 04:05:34.875784 32574 raft_consensus.cc:3037] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:34.875785 376 raft_consensus.cc:3037] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:34.876585 376 raft_consensus.cc:2454] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54b524dbd3a4253a8138f101a109334 in term 1.
I20260927 04:05:34.876785 32574 raft_consensus.cc:2454] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f54b524dbd3a4253a8138f101a109334 in term 1.
I20260927 04:05:34.876776 32646 leader_election.cc:302] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8f46f6bb1b245bd8ec0022b6a213ab8, f54b524dbd3a4253a8138f101a109334; no voters:
I20260927 04:05:34.876968 437 raft_consensus.cc:2781] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:34.877086 437 raft_consensus.cc:686] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 1 LEADER]: Becoming Leader. State: Replica: f54b524dbd3a4253a8138f101a109334, State: Running, Role: LEADER
I20260927 04:05:34.877194 437 consensus_queue.cc:237] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:34.877972 32432 catalog_manager.cc:5495] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 reported cstate change: term changed from 0 to 1, leader changed from <none> to f54b524dbd3a4253a8138f101a109334 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "f54b524dbd3a4253a8138f101a109334" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } health_report { overall_health: UNKNOWN } } }
W20260927 04:05:34.890789 32621 tablet.cc:2369] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:05:34.923429 423 tablet.cc:2369] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:05:34.925347 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.922865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.17.184.194:33059)
0927 04:05:34.923085 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:34.923087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:34.923095 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:05:34.923142 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:34.923143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:34.923160 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:34.923220 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:05:34.923377 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.923972 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.924698 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:34.924877 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:34.924925 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:34.925066 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:05:34.925165 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:05:34.925224 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:34.925261 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 04:05:34.925277 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260927 04:05:34.927158 376 raft_consensus.cc:1261] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 1 FOLLOWER]: Refusing update from remote peer f54b524dbd3a4253a8138f101a109334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:34.927169 32574 raft_consensus.cc:1261] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Refusing update from remote peer f54b524dbd3a4253a8138f101a109334: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:34.927518 439 consensus_queue.cc:1035] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:34.927716 437 consensus_queue.cc:1035] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:34.929859 442 mvcc.cc:205] Tried to move back new op lower bound from 7333814005458923520 to 7333814005257695232. Current Snapshot: MvccSnapshot[applied={T|T < 7333814005458923520}]
I20260927 04:05:34.930131 443 mvcc.cc:205] Tried to move back new op lower bound from 7333814005458923520 to 7333814005257695232. Current Snapshot: MvccSnapshot[applied={T|T < 7333814005458923520}]
I20260927 04:05:34.930159 445 mvcc.cc:205] Tried to move back new op lower bound from 7333814005458923520 to 7333814005257695232. Current Snapshot: MvccSnapshot[applied={T|T < 7333814005458923520}]
I20260927 04:05:34.932307 32432 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:34592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:05:34.932394 32432 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:34592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:05:34.933211 32432 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Sending DeleteTablet for 3 replicas of tablet e82786dd934240a58a240cd000dcfd8f
I20260927 04:05:34.933597 32692 tablet_service.cc:1514] Processing DeleteTablet for tablet e82786dd934240a58a240cd000dcfd8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:34 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:50606
I20260927 04:05:34.933661 356 tablet_service.cc:1514] Processing DeleteTablet for tablet e82786dd934240a58a240cd000dcfd8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:34 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:48116
I20260927 04:05:34.933565 32554 tablet_service.cc:1514] Processing DeleteTablet for tablet e82786dd934240a58a240cd000dcfd8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:34 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:60076
I20260927 04:05:34.933848 452 tablet_replica.cc:330] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: stopping tablet replica
I20260927 04:05:34.933970 454 tablet_replica.cc:330] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: stopping tablet replica
I20260927 04:05:34.933979 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 32470
I20260927 04:05:34.933985 452 raft_consensus.cc:2229] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:05:34.934087 454 raft_consensus.cc:2229] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:05:34.934099 452 raft_consensus.cc:2258] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:34.934274 454 raft_consensus.cc:2258] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:34.934779 452 ts_tablet_manager.cc:1905] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:34.936657 452 ts_tablet_manager.cc:1918] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:05:34.936815 452 log.cc:1196] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/e82786dd934240a58a240cd000dcfd8f
I20260927 04:05:34.937183 452 ts_tablet_manager.cc:1939] T e82786dd934240a58a240cd000dcfd8f P f54b524dbd3a4253a8138f101a109334: Deleting consensus metadata
I20260927 04:05:34.937638 32417 catalog_manager.cc:4841] TS f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059): tablet e82786dd934240a58a240cd000dcfd8f (table test-table [id=b2255942661c433283d8eefe2145a368]) successfully deleted
W20260927 04:05:34.939023 32419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:33715 (error 108)
I20260927 04:05:34.939311 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 32625
I20260927 04:05:34.939679 428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.939557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33715 (local address 127.0.0.1:60082)
0927 04:05:34.939609 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.939640 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 04:05:34.939780 32419 catalog_manager.cc:4584] TS d7c742244c834c469d48bcf7a71f5709 (127.17.184.193:33715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e82786dd934240a58a240cd000dcfd8f: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33715: connect: Connection refused (error 111)
I20260927 04:05:34.943048 453 tablet_replica.cc:330] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: stopping tablet replica
I20260927 04:05:34.943177 453 raft_consensus.cc:2229] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:05:34.943312 453 raft_consensus.cc:2258] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:34.943732 453 ts_tablet_manager.cc:1905] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:34.944500 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 32761
I20260927 04:05:34.948954 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 32399
I20260927 04:05:34.949098 428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:34.948930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44873 (local address 127.0.0.1:48132)
0927 04:05:34.948999 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:34.949056 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.195:44873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 04:05:34.954294 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41965
--webserver_interface=127.17.184.254
--webserver_port=37785
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:35.035843 456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:35.036056 456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:35.036079 456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:35.037271 456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:35.037313 456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:35.037328 456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:35.037340 456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:35.037352 456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:35.038861 456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=37785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:35.039136 456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:35.039386 456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:05:35.044351 456 server_base.cc:1034] running on GCE node
W20260927 04:05:35.044325 464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:35.044317 461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:35.044314 462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:35.044714 456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:35.045011 456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:35.046190 456 hybrid_clock.cc:648] HybridClock initialized: now 1790481935046151 us; error 64 us; skew 500 ppm
Sep 27 04:05:35 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:35.049176 456 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:05:35.049674 456 webserver.cc:460] Webserver started at http://127.17.184.254:37785/ using document root <none> and password file <none>
I20260927 04:05:35.049922 456 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:35.049990 456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:35.051771 456 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:05:35.052603 471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:35.052811 456 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:35.052918 456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "365d53ef8dc548d0a6ec08f277478b10"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "a66c0c86a0ad6b65ee92981755dd1e68"
server_key_iv: "24915451f998833ab1cfd31d2c78aa2e"
server_key_version: "encryptionkey@0"
I20260927 04:05:35.053274 456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:35.060293 456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:35.062484 456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:35.062683 456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:35.067107 456 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:41965
I20260927 04:05:35.067130 523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:41965 every 8 connection(s)
I20260927 04:05:35.067625 456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:05:35.069293 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 456
I20260927 04:05:35.069638 524 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:05:35.069660 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:33715
--local_ip_for_outbound_sockets=127.17.184.193
--tserver_master_addrs=127.17.184.254:41965
--webserver_port=40637
--webserver_interface=127.17.184.193
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:35.070876 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Bootstrap starting.
I20260927 04:05:35.080154 524 log.cc:824] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:35.082765 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:05:35.083127 524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10: Bootstrap complete.
I20260927 04:05:35.085574 524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:35.085809 524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365d53ef8dc548d0a6ec08f277478b10, State: Initialized, Role: FOLLOWER
I20260927 04:05:35.085992 524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:35.086067 524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:05:35.086122 524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:05:35.086174 524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 2 FOLLOWER]: Advancing to term 3
I20260927 04:05:35.087319 524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:35.087410 524 leader_election.cc:302] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 365d53ef8dc548d0a6ec08f277478b10; no voters:
I20260927 04:05:35.087671 524 leader_election.cc:288] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 04:05:35.087710 528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 04:05:35.087821 528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [term 3 LEADER]: Becoming Leader. State: Replica: 365d53ef8dc548d0a6ec08f277478b10, State: Running, Role: LEADER
I20260927 04:05:35.087966 524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:35.087975 528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } }
I20260927 04:05:35.088301 530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365d53ef8dc548d0a6ec08f277478b10. Latest consensus state: current_term: 3 leader_uuid: "365d53ef8dc548d0a6ec08f277478b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } } }
I20260927 04:05:35.088305 529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "365d53ef8dc548d0a6ec08f277478b10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365d53ef8dc548d0a6ec08f277478b10" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41965 } } }
I20260927 04:05:35.088481 530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:35.088754 529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365d53ef8dc548d0a6ec08f277478b10 [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:35.089210 536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:05:35.090159 536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:05:35.090440 536 catalog_manager.cc:1259] Loaded cluster ID: ca1be0ff67034f1482a26e3026c4f4f9
I20260927 04:05:35.090570 536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 04:05:35.090806 536 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f42925349ad google::LogMessage::Fail() at ??:0
@ 0x7f429253788a google::LogMessage::SendToLog() at ??:0
@ 0x7f42925344de google::LogMessage::Flush() at ??:0
@ 0x7f4292535a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f429a4590f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f429a4628b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f42944020d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f42943f7954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f429525b6db start_thread at ??:0
@ 0x7f429103261f clone at ??:0
W20260927 04:05:35.166689 526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:35.166968 526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:35.167021 526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:35.168236 526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:35.168308 526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:35.168356 526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:05:35.169884 526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:33715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=40637
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:35.170205 526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:35.170487 526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:35.175824 563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:35.175854 561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:35.175845 560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:35.176206 526 server_base.cc:1034] running on GCE node
I20260927 04:05:35.176429 526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:35.176723 526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:35.177886 526 hybrid_clock.cc:648] HybridClock initialized: now 1790481935177869 us; error 34 us; skew 500 ppm
Sep 27 04:05:35 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:35.180577 526 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:05:35.181067 526 webserver.cc:460] Webserver started at http://127.17.184.193:40637/ using document root <none> and password file <none>
I20260927 04:05:35.181314 526 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:35.181384 526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:35.182787 526 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:35.183561 570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:35.183735 526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:05:35.183840 526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "d7c742244c834c469d48bcf7a71f5709"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "6018527564250021b5d017e42b5da7a3"
server_key_iv: "be688a31aac4b2c5b1d608873df468fd"
server_key_version: "encryptionkey@0"
I20260927 04:05:35.184193 526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:35.199925 526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:35.202308 526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:35.202451 526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:35.202697 526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:35.203248 577 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:05:35.204751 526 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:05:35.204802 526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:05:35.204836 526 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:05:35.205507 526 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:05:35.205552 526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:05:35.205586 577 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Bootstrap starting.
I20260927 04:05:35.213239 526 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:33715
I20260927 04:05:35.213457 684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:33715 every 8 connection(s)
I20260927 04:05:35.213804 526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:05:35.215257 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 526
I20260927 04:05:35.215526 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:33059
--local_ip_for_outbound_sockets=127.17.184.194
--tserver_master_addrs=127.17.184.254:41965
--webserver_port=35113
--webserver_interface=127.17.184.194
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:35.216727 577 log.cc:824] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:35.219693 577 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:05:35.220038 577 tablet_bootstrap.cc:492] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Bootstrap complete.
I20260927 04:05:35.221308 577 ts_tablet_manager.cc:1397] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260927 04:05:35.223214 577 raft_consensus.cc:348] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:35.223623 577 raft_consensus.cc:729] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c742244c834c469d48bcf7a71f5709, State: Initialized, Role: FOLLOWER
I20260927 04:05:35.223817 577 consensus_queue.cc:260] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:35.224144 577 ts_tablet_manager.cc:1428] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
W20260927 04:05:35.276381 687 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:05:35.214516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:39805)
0927 04:05:35.214688 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.214745 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:35.215439 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:35.276159 (+ 60720us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
W20260927 04:05:35.276683 685 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:41965 (0 consecutive failures): Network error: Failed to ping master at 127.17.184.254:41965: Client connection negotiation failed: client connection to 127.17.184.254:41965: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 04:05:35.277063 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:35.276938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:47721)
0927 04:05:35.276997 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.277042 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:05:35.277482 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:35.277349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:56049)
0927 04:05:35.277424 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.277458 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 04:05:35.277665 685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:05:35.301084 688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:35.301282 688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:35.301306 688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:35.302423 688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:35.302459 688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:35.302484 688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:05:35.303952 688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:33059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=35113
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:35.304137 688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:35.304313 688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:35.309537 698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:35.309545 696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:35.309711 695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:35.309857 688 server_base.cc:1034] running on GCE node
I20260927 04:05:35.310120 688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:35.310367 688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:35.311509 688 hybrid_clock.cc:648] HybridClock initialized: now 1790481935311499 us; error 33 us; skew 500 ppm
Sep 27 04:05:35 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:35.313963 688 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:05:35.314407 688 webserver.cc:460] Webserver started at http://127.17.184.194:35113/ using document root <none> and password file <none>
I20260927 04:05:35.314657 688 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:35.314706 688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:35.316100 688 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:35.316769 705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:35.316977 688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:35.317088 688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "f54b524dbd3a4253a8138f101a109334"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "0994a6af09b6a9869922f27c898588bc"
server_key_iv: "8a998b4fd6054b922c094bd9b45bcb1a"
server_key_version: "encryptionkey@0"
I20260927 04:05:35.317471 688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:35.340667 688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:35.342756 688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:35.342872 688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:35.343204 688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:35.343561 688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:35.343621 688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:35.343654 688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:35.343681 688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:35.349747 688 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33059
I20260927 04:05:35.349788 818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33059 every 8 connection(s)
I20260927 04:05:35.350132 688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:05:35.351233 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:35.350804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:33509)
0927 04:05:35.351011 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.351091 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":118,"threads_started":1}
I20260927 04:05:35.351653 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 688
W20260927 04:05:35.351742 819 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:41965 (0 consecutive failures): Network error: Failed to ping master at 127.17.184.254:41965: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
I20260927 04:05:35.351980 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:44873
--local_ip_for_outbound_sockets=127.17.184.195
--tserver_master_addrs=127.17.184.254:41965
--webserver_port=39613
--webserver_interface=127.17.184.195
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:46817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:05:35.352031 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:35.351924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:42853)
0927 04:05:35.351975 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.352000 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 04:05:35.352573 821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:35.352402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:52165)
0927 04:05:35.352471 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.352540 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 04:05:35.352669 819 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:05:35.438331 822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:35.438536 822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:35.438560 822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:35.439832 822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:35.439874 822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:35.439900 822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:05:35.441386 822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:46817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:44873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=39613
--tserver_master_addrs=127.17.184.254:41965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:35.441627 822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:35.441864 822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:35.447101 827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:35.447162 830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:35.447113 828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:35.447322 822 server_base.cc:1034] running on GCE node
I20260927 04:05:35.447644 822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:35.447897 822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:35.449043 822 hybrid_clock.cc:648] HybridClock initialized: now 1790481935449029 us; error 29 us; skew 500 ppm
Sep 27 04:05:35 dist-test-slave-zqrn krb5kdc[31850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:35.451756 822 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:05:35.452219 822 webserver.cc:460] Webserver started at http://127.17.184.195:39613/ using document root <none> and password file <none>
I20260927 04:05:35.452415 822 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:35.452457 822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:35.453811 822 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:35.454473 837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:35.454747 822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:35.454849 822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8"
format_stamp: "Formatted at 2026-09-27 04:05:33 on dist-test-slave-zqrn"
server_key: "25b1e9f344c9804e83c4e6a2f5058d27"
server_key_iv: "83f09d3a037c62cba51702ccd3732ac5"
server_key_version: "encryptionkey@0"
I20260927 04:05:35.455271 822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:35.472630 822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:35.474843 822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:35.475046 822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:35.475312 822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:35.475785 844 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 04:05:35.477387 844 ts_tablet_manager.cc:1852] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 04:05:35.477768 844 ts_tablet_manager.cc:1905] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:35.480057 844 ts_tablet_manager.cc:1918] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:05:35.480162 844 log.cc:1196] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/e82786dd934240a58a240cd000dcfd8f
I20260927 04:05:35.480458 844 ts_tablet_manager.cc:1939] T e82786dd934240a58a240cd000dcfd8f P a8f46f6bb1b245bd8ec0022b6a213ab8: Deleting consensus metadata
I20260927 04:05:35.480728 822 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 04:05:35.480810 822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260927 04:05:35.480867 822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:35.480892 822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:35.487046 822 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:44873
I20260927 04:05:35.487093 951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:44873 every 8 connection(s)
I20260927 04:05:35.487389 822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:05:35.487835 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 822
I20260927 04:05:35.488545 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:35.488141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:38793)
0927 04:05:35.488318 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.488396 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
W20260927 04:05:35.488787 952 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:41965 (0 consecutive failures): Network error: Failed to ping master at 127.17.184.254:41965: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
I20260927 04:05:35.489070 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:35.488982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:46747)
0927 04:05:35.489028 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.489053 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 04:05:35.489315 955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:35.489264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:40931)
0927 04:05:35.489284 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:35.489302 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260927 04:05:35.489406 952 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 04:05:36.278344 956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:36.278095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:37925)
0927 04:05:36.278240 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:36.278282 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260927 04:05:36.353207 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:36.352973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:48525)
0927 04:05:36.353117 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:36.353156 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":55,"threads_started":1}
I20260927 04:05:36.489976 958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:36.489764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:47993)
0927 04:05:36.489902 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:36.489941 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
I20260927 04:05:36.990628 959 raft_consensus.cc:482] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:36.990758 959 raft_consensus.cc:504] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:36.991133 959 leader_election.cc:288] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059), a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873)
Sep 27 04:05:36 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.194@KRBTEST.COM
Sep 27 04:05:36 dist-test-slave-zqrn krb5kdc[31850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481935, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.195@KRBTEST.COM
I20260927 04:05:37.002547 962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:36.991597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44873 (local address 127.17.184.193:41113)
0927 04:05:36.991707 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:36.991716 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:36.991760 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:36.992835 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:36.992836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:36.993071 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:36.993248 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:36.993254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:36.994344 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:36.994346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:36.995342 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:36.995345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:36.995963 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:36.995964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:36.996097 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:36.996698 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:36.996706 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:36.998561 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:37.001026 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:37.001027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:37.001028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:37.001228 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:37.001489 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:37.001490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:37.001491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:37.001614 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:37.002022 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:37.002029 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:37.002152 (+ 123us) client_negotiation.cc:769] Sending connection context
0927 04:05:37.002255 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 04:05:37.002439 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":40,"thread_start_us":45,"threads_started":1}
I20260927 04:05:37.002655 961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:36.991496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:58581 (local address 127.17.184.194:33059)
0927 04:05:36.991681 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:36.991694 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:36.991722 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:05:36.992387 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:36.992390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:36.992438 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:36.992572 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:36.993375 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:36.994329 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:36.995448 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:36.995894 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:36.996036 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:36.998647 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:36.998652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:36.998652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:36.998664 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:37.000765 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:37.001303 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:37.001307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:37.001307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:37.001411 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:37.001702 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:37.001703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:37.001703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:37.001946 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:37.002047 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:37.002386 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 04:05:37.002534 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":59,"threads_started":1}
I20260927 04:05:37.002841 963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:36.991755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:41113 (local address 127.17.184.195:44873)
0927 04:05:36.992024 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:36.992059 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:36.992091 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:05:36.992604 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:36.992608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:36.992697 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:36.992838 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:36.993376 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:36.994236 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:36.995478 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:36.995868 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:36.995949 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:36.998699 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:36.998702 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:36.998703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:36.998713 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:37.000821 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:37.001322 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:37.001326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:37.001326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:37.001417 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:37.001701 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:37.001702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:37.001702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:37.001946 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:37.002044 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:37.002444 (+ 400us) server_negotiation.cc:299] Negotiation successful
0927 04:05:37.002574 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":52,"threads_started":1}
I20260927 04:05:37.003121 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:36.991386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33059 (local address 127.17.184.193:58581)
0927 04:05:36.991513 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:36.991527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:36.991583 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:36.992640 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:36.992643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:36.993058 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:36.993248 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:36.993254 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:36.994435 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:36.994436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:36.995343 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:36.995346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:36.996009 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:36.996011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:36.996094 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:36.996719 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:36.996723 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:36.998221 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:37.000882 (+ 2661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:37.000890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:37.000892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:37.001207 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:37.001489 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:37.001490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:37.001490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:37.001618 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:37.002026 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:37.002032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:37.002161 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 04:05:37.002253 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 04:05:37.002994 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":602,"thread_start_us":81,"threads_started":1}
W20260927 04:05:37.003684 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
W20260927 04:05:37.003859 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
I20260927 04:05:37.003937 572 leader_election.cc:302] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c742244c834c469d48bcf7a71f5709; no voters: a8f46f6bb1b245bd8ec0022b6a213ab8, f54b524dbd3a4253a8138f101a109334
I20260927 04:05:37.004142 959 raft_consensus.cc:2726] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:05:37.279174 960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:37.279020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:56265)
0927 04:05:37.279086 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:37.279123 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 04:05:37.353905 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:37.353675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:34157)
0927 04:05:37.353804 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:37.353853 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260927 04:05:37.490759 965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:37.490443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:44045)
0927 04:05:37.490673 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:37.490710 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":125,"threads_started":1}
I20260927 04:05:38.280001 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:38.279746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:46463)
0927 04:05:38.279909 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:38.279953 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260927 04:05:38.354665 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:38.354427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:51675)
0927 04:05:38.354571 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:38.354616 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":71,"threads_started":1}
I20260927 04:05:38.491643 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:38.491354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:46585)
0927 04:05:38.491557 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:38.491599 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":73,"threads_started":1}
I20260927 04:05:38.527441 969 raft_consensus.cc:482] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:38.527575 969 raft_consensus.cc:504] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:38.527761 969 leader_election.cc:288] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059), a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873)
W20260927 04:05:38.528344 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
W20260927 04:05:38.528414 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
I20260927 04:05:38.528435 572 leader_election.cc:302] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c742244c834c469d48bcf7a71f5709; no voters: a8f46f6bb1b245bd8ec0022b6a213ab8, f54b524dbd3a4253a8138f101a109334
I20260927 04:05:38.528527 969 raft_consensus.cc:2726] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:05:39.280876 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:39.280591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:58719)
0927 04:05:39.280768 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:39.280818 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":72,"threads_started":1}
I20260927 04:05:39.355393 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:39.355187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:39819)
0927 04:05:39.355311 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:39.355351 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260927 04:05:39.492511 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:39.492274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:39775)
0927 04:05:39.492408 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:39.492451 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260927 04:05:40.281661 973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:40.281435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:60809)
0927 04:05:40.281576 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:40.281615 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260927 04:05:40.356127 974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:40.355889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:55601)
0927 04:05:40.356034 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:40.356077 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260927 04:05:40.493314 975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:40.493093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:49271)
0927 04:05:40.493231 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:40.493270 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260927 04:05:41.282598 976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:41.282320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:44229)
0927 04:05:41.282495 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:41.282547 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260927 04:05:41.343866 977 raft_consensus.cc:482] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:41.344012 977 raft_consensus.cc:504] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:41.344185 977 leader_election.cc:288] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059), a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873)
W20260927 04:05:41.344700 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
W20260927 04:05:41.344770 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
I20260927 04:05:41.344785 572 leader_election.cc:302] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c742244c834c469d48bcf7a71f5709; no voters: a8f46f6bb1b245bd8ec0022b6a213ab8, f54b524dbd3a4253a8138f101a109334
I20260927 04:05:41.344978 977 raft_consensus.cc:2726] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:05:41.356814 978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:41.356595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:38733)
0927 04:05:41.356731 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:41.356772 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":59,"threads_started":1}
I20260927 04:05:41.494051 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:41.493833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:36313)
0927 04:05:41.493967 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:41.494007 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260927 04:05:42.283476 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:42.283226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:59655)
0927 04:05:42.283386 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:42.283428 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":68,"threads_started":1}
I20260927 04:05:42.357492 981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:42.357243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:34861)
0927 04:05:42.357406 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:42.357448 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":71,"threads_started":1}
I20260927 04:05:42.494798 982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:42.494552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:43081)
0927 04:05:42.494705 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:42.494754 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260927 04:05:43.079813 983 raft_consensus.cc:482] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:43.079923 983 raft_consensus.cc:504] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:43.080119 983 leader_election.cc:288] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059), a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873)
W20260927 04:05:43.080549 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
W20260927 04:05:43.080673 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
I20260927 04:05:43.080691 572 leader_election.cc:302] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c742244c834c469d48bcf7a71f5709; no voters: a8f46f6bb1b245bd8ec0022b6a213ab8, f54b524dbd3a4253a8138f101a109334
I20260927 04:05:43.080771 983 raft_consensus.cc:2726] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:05:43.284340 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:43.284093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:56179)
0927 04:05:43.284251 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:43.284293 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260927 04:05:43.358377 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:43.358051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:34419)
0927 04:05:43.358258 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:43.358306 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":65,"threads_started":1}
I20260927 04:05:43.495656 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:43.495387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:33935)
0927 04:05:43.495527 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:43.495611 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260927 04:05:44.285162 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:44.284913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:42633)
0927 04:05:44.285068 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:44.285115 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260927 04:05:44.359282 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:44.359042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:52037)
0927 04:05:44.359185 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:44.359231 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260927 04:05:44.496462 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:44.496204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:35467)
0927 04:05:44.496366 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:44.496414 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":65,"threads_started":1}
I20260927 04:05:45.286024 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:45.285762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:58793)
0927 04:05:45.285933 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:45.285977 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":67,"threads_started":1}
I20260927 04:05:45.360190 991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:45.359939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:53029)
0927 04:05:45.360092 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:45.360143 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":63,"threads_started":1}
I20260927 04:05:45.497282 992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:45.497071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:39615)
0927 04:05:45.497183 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:45.497217 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260927 04:05:46.287071 993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:46.286721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:48089)
0927 04:05:46.286958 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:46.287017 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
I20260927 04:05:46.361085 994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:46.360832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:44933)
0927 04:05:46.360991 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:46.361037 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260927 04:05:46.498095 995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:46.497843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:51687)
0927 04:05:46.498001 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:46.498045 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":72,"threads_started":1}
I20260927 04:05:47.090137 996 raft_consensus.cc:482] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:47.090243 996 raft_consensus.cc:504] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f54b524dbd3a4253a8138f101a109334" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33059 } } peers { permanent_uuid: "a8f46f6bb1b245bd8ec0022b6a213ab8" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44873 } } peers { permanent_uuid: "d7c742244c834c469d48bcf7a71f5709" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33715 } }
I20260927 04:05:47.090595 996 leader_election.cc:288] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059), a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873)
W20260927 04:05:47.091113 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a8f46f6bb1b245bd8ec0022b6a213ab8 (127.17.184.195:44873): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
W20260927 04:05:47.091223 572 leader_election.cc:341] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f54b524dbd3a4253a8138f101a109334 (127.17.184.194:33059): Not found: Tablet not found: e82786dd934240a58a240cd000dcfd8f
I20260927 04:05:47.091253 572 leader_election.cc:302] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7c742244c834c469d48bcf7a71f5709; no voters: a8f46f6bb1b245bd8ec0022b6a213ab8, f54b524dbd3a4253a8138f101a109334
I20260927 04:05:47.091338 996 raft_consensus.cc:2726] T e82786dd934240a58a240cd000dcfd8f P d7c742244c834c469d48bcf7a71f5709 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:05:47.287819 997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:47.287563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:51865)
0927 04:05:47.287718 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:47.287770 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
I20260927 04:05:47.361894 998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:47.361660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:52005)
0927 04:05:47.361809 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:47.361847 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260927 04:05:47.499010 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:47.498691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:55225)
0927 04:05:47.498846 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:47.498891 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260927 04:05:48.288839 1000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:48.288555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:33629)
0927 04:05:48.288725 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:48.288792 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":67,"threads_started":1}
I20260927 04:05:48.362737 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:48.362469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:50911)
0927 04:05:48.362639 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:48.362689 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260927 04:05:48.500025 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:48.499749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:52397)
0927 04:05:48.499916 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:48.499970 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260927 04:05:49.289660 1003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:49.289396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:38317)
0927 04:05:49.289544 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:49.289599 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":62,"threads_started":1}
I20260927 04:05:49.363657 1004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:49.363402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:60343)
0927 04:05:49.363560 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:49.363605 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260927 04:05:49.500859 1005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:49.500604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.195:36037)
0927 04:05:49.500765 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:49.500810 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
I20260927 04:05:50.290439 1006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:50.290228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.193:50565)
0927 04:05:50.290354 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:50.290390 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260927 04:05:50.364392 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:50.364158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41965 (local address 127.17.184.194:34207)
0927 04:05:50.364306 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:50.364343 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:41965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260927 04:05:50.490980 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 526
I20260927 04:05:50.495788 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 688
I20260927 04:05:50.500385 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 822
2026-09-27T04:05:50Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18721 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:05:50 dist-test-slave-zqrn krb5kdc[1010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:05:50 dist-test-slave-zqrn krb5kdc[1010](info): set up 2 sockets
Sep 27 04:05:50 dist-test-slave-zqrn krb5kdc[1010](info): commencing operation
krb5kdc: starting...
W20260927 04:05:51.575748 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:05:51 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:05:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:05:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:05:51.700011 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44953
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:51.784197 1023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:51.784451 1023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:51.784473 1023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:51.785583 1023 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 04:05:51.785617 1023 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 04:05:51.785630 1023 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 04:05:51.785642 1023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:05:51.785655 1023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:51.785674 1023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:51.785686 1023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:05:51.785698 1023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:05:51.787104 1023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:51.787290 1023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:51.787456 1023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:51.792421 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:51.792479 1028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:51.792613 1023 server_base.cc:1034] running on GCE node
W20260927 04:05:51.792435 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:51.792938 1023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:51.793242 1023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:51.794418 1023 hybrid_clock.cc:648] HybridClock initialized: now 1790481951794411 us; error 37 us; skew 500 ppm
Sep 27 04:05:51 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481951, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:51.796959 1023 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:05:51.797396 1023 webserver.cc:460] Webserver started at http://127.17.184.254:33353/ using document root <none> and password file <none>
I20260927 04:05:51.797617 1023 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:51.797694 1023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:51.797847 1023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:51.798959 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "a0233c61cf5e3c43f625c488551bcd86"
server_key_iv: "2f44e17c050daf890f7746783147ac49"
server_key_version: "encryptionkey@0"
I20260927 04:05:51.799348 1023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "a0233c61cf5e3c43f625c488551bcd86"
server_key_iv: "2f44e17c050daf890f7746783147ac49"
server_key_version: "encryptionkey@0"
I20260927 04:05:51.800741 1023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:05:51.801493 1038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:51.801703 1023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:05:51.801810 1023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "a0233c61cf5e3c43f625c488551bcd86"
server_key_iv: "2f44e17c050daf890f7746783147ac49"
server_key_version: "encryptionkey@0"
I20260927 04:05:51.801898 1023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:51.810001 1023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:51.812124 1023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:51.812307 1023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:51.816504 1023 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44953
I20260927 04:05:51.816536 1090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44953 every 8 connection(s)
I20260927 04:05:51.816913 1023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:05:51.817752 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:51.820132 1091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Bootstrap starting.
I20260927 04:05:51.820731 1091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:51.821012 1091 log.cc:824] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:51.821712 1091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: No bootstrap required, opened a new log
I20260927 04:05:51.823293 1091 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:05:51.823429 1091 raft_consensus.cc:374] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:51.823452 1091 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff842ae9f3b4bd2a33cf37bd61d6a8d, State: Initialized, Role: FOLLOWER
I20260927 04:05:51.823588 1091 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:05:51.823668 1091 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:05:51.823694 1091 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:05:51.823724 1091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:51.824206 1091 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:05:51.824287 1091 leader_election.cc:302] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff842ae9f3b4bd2a33cf37bd61d6a8d; no voters:
I20260927 04:05:51.824411 1091 leader_election.cc:288] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:05:51.824462 1094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:51.824693 1091 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:05:51.824676 1094 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 1 LEADER]: Becoming Leader. State: Replica: dff842ae9f3b4bd2a33cf37bd61d6a8d, State: Running, Role: LEADER
I20260927 04:05:51.824855 1094 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:05:51.824928 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 1023
I20260927 04:05:51.825057 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:05:51.825270 18147 external_mini_cluster.cc:1428] Setting key 8a09164be5741669dc0feea27f31e7ac
I20260927 04:05:51.825505 1095 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } } }
I20260927 04:05:51.825600 1095 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: This master's current role is: LEADER
I20260927 04:05:51.826045 1096 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff842ae9f3b4bd2a33cf37bd61d6a8d. Latest consensus state: current_term: 1 leader_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } } }
I20260927 04:05:51.826143 1096 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: This master's current role is: LEADER
W20260927 04:05:51.826871 1111 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:05:51.826994 1111 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:05:51.827066 1105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:05:51.827240 1105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:05:51.828653 1105 catalog_manager.cc:1347] Generated new cluster ID: d76cfbdc2ba4425c80d8a804c91427ce
I20260927 04:05:51.828708 1105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:05:51 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:51.837237 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:51.825776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.17.184.254:44953)
0927 04:05:51.826064 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:51.826075 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:51.826115 (+ 40us) server_negotiation.cc:407] Connection header received
0927 04:05:51.826250 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:51.826258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:51.826322 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:51.826478 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:51.827394 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:51.828774 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:51.829829 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:51.830458 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:51.830589 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:51.833509 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:51.833513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:51.833514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:51.833526 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:51.835553 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:51.835974 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:51.835977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:51.835977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:51.836081 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:51.836298 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:51.836299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:51.836299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:51.836592 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:51.836737 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:51.836876 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 04:05:51.837010 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":61,"threads_started":1}
I20260927 04:05:51.838380 1105 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:05:51.838883 1105 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:05:51.847684 1105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Generated new TSK 0
I20260927 04:05:51.847894 1105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:05:51.883810 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44953
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:51.966575 1117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:51.966796 1117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:51.966820 1117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:51.968107 1117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:51.968155 1117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:51.968183 1117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:05:51.969676 1117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:51.969936 1117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:51.970180 1117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:51.975287 1125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:51.975260 1122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:51.975600 1117 server_base.cc:1034] running on GCE node
W20260927 04:05:51.975275 1123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:51.975871 1117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:51.976136 1117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:51.977311 1117 hybrid_clock.cc:648] HybridClock initialized: now 1790481951977306 us; error 36 us; skew 500 ppm
Sep 27 04:05:51 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481951, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:51.979836 1117 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:05:51.980296 1117 webserver.cc:460] Webserver started at http://127.17.184.193:40785/ using document root <none> and password file <none>
I20260927 04:05:51.980521 1117 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:51.980588 1117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:51.980751 1117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:51.981853 1117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "02d7a3620f864ca0bb7cd38b31c0f427"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "9f9f72dd24e49bc50d154f85377ff794"
server_key_iv: "c8d44fe05389f355ca004c5feead5f72"
server_key_version: "encryptionkey@0"
I20260927 04:05:51.982246 1117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "02d7a3620f864ca0bb7cd38b31c0f427"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "9f9f72dd24e49bc50d154f85377ff794"
server_key_iv: "c8d44fe05389f355ca004c5feead5f72"
server_key_version: "encryptionkey@0"
I20260927 04:05:51.983809 1117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:05:51.984710 1132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:51.985028 1117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:05:51.985138 1117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "02d7a3620f864ca0bb7cd38b31c0f427"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "9f9f72dd24e49bc50d154f85377ff794"
server_key_iv: "c8d44fe05389f355ca004c5feead5f72"
server_key_version: "encryptionkey@0"
I20260927 04:05:51.985214 1117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:52.004001 1117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:52.006361 1117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:52.006538 1117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:52.006809 1117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:52.007167 1117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:52.007236 1117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:52.007280 1117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:52.007305 1117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:52.013531 1117 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:35897
I20260927 04:05:52.013578 1245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:35897 every 8 connection(s)
I20260927 04:05:52.013914 1117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:05:52.019481 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 1117
I20260927 04:05:52.019580 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:05:52.019825 18147 external_mini_cluster.cc:1428] Setting key b5b558f70eceb1ef273f65af1d55ddbe
Sep 27 04:05:52 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481951, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:52.025120 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.014767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:47619 (local address 127.17.184.254:44953)
0927 04:05:52.014824 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.014827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.015011 (+ 184us) server_negotiation.cc:407] Connection header received
0927 04:05:52.015586 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.015588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.015606 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.015675 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:52.016746 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.017272 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.018328 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.018540 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.018623 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.021652 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:52.021653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:52.021654 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:52.021660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:52.023356 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:52.023856 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:52.023857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:52.023857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:52.023908 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:52.024272 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:52.024273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:52.024273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:52.024432 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:52.024509 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.024891 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.024994 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 04:05:52.025331 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.014631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:47619)
0927 04:05:52.014841 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.014906 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:52.015452 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:52.015753 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:52.015757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:52.016412 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:52.016627 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.016632 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.017376 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.017377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.018216 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.018218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.018636 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.018638 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.018784 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.019469 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:52.019478 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:52.021530 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:52.023438 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:52.023446 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:52.023449 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:52.023777 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:52.023979 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:52.023980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:52.023980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:52.024137 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:52.024509 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:52.024516 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:52.024706 (+ 190us) client_negotiation.cc:769] Sending connection context
0927 04:05:52.024839 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 04:05:52.025036 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
I20260927 04:05:52.025913 1246 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44953
I20260927 04:05:52.026077 1246 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:52.026278 1246 heartbeater.cc:511] Master 127.17.184.254:44953 requested a full tablet report, sending...
I20260927 04:05:52.027016 1055 ts_manager.cc:194] Registered new tserver with Master: 02d7a3620f864ca0bb7cd38b31c0f427 (127.17.184.193:35897)
I20260927 04:05:52.027971 1055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:47619
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:05:52.051613 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44953
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:52.133015 1251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:52.133229 1251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:52.133253 1251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:52.134418 1251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:52.134462 1251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:52.134490 1251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:05:52.135962 1251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:52.136234 1251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:52.136476 1251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:52.141610 1256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:52.141721 1257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:52.141618 1259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:52.141971 1251 server_base.cc:1034] running on GCE node
I20260927 04:05:52.142246 1251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:52.142484 1251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:52.143656 1251 hybrid_clock.cc:648] HybridClock initialized: now 1790481952143626 us; error 52 us; skew 500 ppm
Sep 27 04:05:52 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481952, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:52.146267 1251 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:05:52.146744 1251 webserver.cc:460] Webserver started at http://127.17.184.194:39155/ using document root <none> and password file <none>
I20260927 04:05:52.147027 1251 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:52.147078 1251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:52.147202 1251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:52.148272 1251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "0ed9147adf4c432db5f4fcf65aab4a20"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "73092d704412955dd5469ebc08935a74"
server_key_iv: "c3efeb6ea77d5391485e4748c8427e76"
server_key_version: "encryptionkey@0"
I20260927 04:05:52.148623 1251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "0ed9147adf4c432db5f4fcf65aab4a20"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "73092d704412955dd5469ebc08935a74"
server_key_iv: "c3efeb6ea77d5391485e4748c8427e76"
server_key_version: "encryptionkey@0"
I20260927 04:05:52.149966 1251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:05:52.150750 1266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:52.150969 1251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:05:52.151065 1251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "0ed9147adf4c432db5f4fcf65aab4a20"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "73092d704412955dd5469ebc08935a74"
server_key_iv: "c3efeb6ea77d5391485e4748c8427e76"
server_key_version: "encryptionkey@0"
I20260927 04:05:52.151150 1251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:52.174088 1251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:52.176405 1251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:52.176577 1251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:52.176821 1251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:52.177137 1251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:52.177199 1251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:52.177235 1251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:52.177264 1251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:52.183324 1251 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33023
I20260927 04:05:52.183372 1379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33023 every 8 connection(s)
I20260927 04:05:52.183686 1251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:05:52.186749 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 1251
I20260927 04:05:52.186865 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:05:52.187114 18147 external_mini_cluster.cc:1428] Setting key 5923075a6e38bf77ff6cb49622b9705e
Sep 27 04:05:52 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481952, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:52.193965 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.184399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:36107 (local address 127.17.184.254:44953)
0927 04:05:52.184461 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.184463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.184649 (+ 186us) server_negotiation.cc:407] Connection header received
0927 04:05:52.185111 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.185112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.185121 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.185181 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:52.186091 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.186702 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.188147 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.188482 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.188561 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.191086 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:52.191087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:52.191088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:52.191094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:52.192463 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:52.192971 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:52.192971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:52.192972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:52.193024 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:52.193324 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:52.193325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:52.193325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:52.193422 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:52.193471 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.193827 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.193899 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 04:05:52.194254 1382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.184356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:36107)
0927 04:05:52.184516 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.184556 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:52.185031 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:52.185233 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:52.185236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:52.185781 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:52.185974 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.185978 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.186804 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.186805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.188061 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.188065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.188567 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.188568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.188670 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.189281 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:52.189289 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:52.190983 (+ 1694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:52.192548 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:52.192556 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:52.192558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:52.192892 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:52.193093 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:52.193094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:52.193094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:52.193254 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:52.193487 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:52.193493 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:52.193676 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 04:05:52.193810 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 04:05:52.193992 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260927 04:05:52.194808 1380 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44953
I20260927 04:05:52.194943 1380 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:52.195130 1380 heartbeater.cc:511] Master 127.17.184.254:44953 requested a full tablet report, sending...
I20260927 04:05:52.195560 1055 ts_manager.cc:194] Registered new tserver with Master: 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023)
I20260927 04:05:52.195933 1055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:36107
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:05:52.214776 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44953
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:05:52.295557 1385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:05:52.295799 1385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:05:52.295836 1385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:05:52.296988 1385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:05:52.297057 1385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:05:52.297103 1385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:05:52.298585 1385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1385
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:05:52.298859 1385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:05:52.299109 1385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:05:52.304759 1390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:52.304760 1391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:05:52.304987 1393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:05:52.304852 1385 server_base.cc:1034] running on GCE node
I20260927 04:05:52.305251 1385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:05:52.305513 1385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:05:52.306677 1385 hybrid_clock.cc:648] HybridClock initialized: now 1790481952306673 us; error 37 us; skew 500 ppm
Sep 27 04:05:52 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481952, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:05:52.309405 1385 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:05:52.309912 1385 webserver.cc:460] Webserver started at http://127.17.184.195:35745/ using document root <none> and password file <none>
I20260927 04:05:52.310154 1385 fs_manager.cc:359] Metadata directory not provided
I20260927 04:05:52.310225 1385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:05:52.310382 1385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:05:52.311573 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "12ab387c486444ebb27c57f6f83e578b"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "5e6038173d98851baea9e666b8bf895b"
server_key_iv: "c09bfa6eacee44daf37df46878976c19"
server_key_version: "encryptionkey@0"
I20260927 04:05:52.311944 1385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "12ab387c486444ebb27c57f6f83e578b"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "5e6038173d98851baea9e666b8bf895b"
server_key_iv: "c09bfa6eacee44daf37df46878976c19"
server_key_version: "encryptionkey@0"
I20260927 04:05:52.313314 1385 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260927 04:05:52.314169 1400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:52.314395 1385 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:05:52.314489 1385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "12ab387c486444ebb27c57f6f83e578b"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "5e6038173d98851baea9e666b8bf895b"
server_key_iv: "c09bfa6eacee44daf37df46878976c19"
server_key_version: "encryptionkey@0"
I20260927 04:05:52.314579 1385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:05:52.321739 1385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:05:52.324043 1385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:05:52.324203 1385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:05:52.324430 1385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:05:52.324786 1385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:05:52.324838 1385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:52.324880 1385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:05:52.324908 1385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:05:52.331100 1385 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:34793
I20260927 04:05:52.331146 1513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:34793 every 8 connection(s)
I20260927 04:05:52.331467 1385 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:05:52 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481952, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:05:52.339733 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 1385
I20260927 04:05:52.339818 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:05:52.339995 18147 external_mini_cluster.cc:1428] Setting key 744a123d17b2af318483cc4c9295a371
I20260927 04:05:52.341471 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.332191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:41291 (local address 127.17.184.254:44953)
0927 04:05:52.332276 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.332279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.332428 (+ 149us) server_negotiation.cc:407] Connection header received
0927 04:05:52.333022 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.333024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.333039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.333100 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:52.334038 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.334556 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.335668 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.335889 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.335977 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.338283 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:52.338284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:52.338284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:52.338293 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:52.339873 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:52.340378 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:52.340380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:52.340381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:52.340471 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:52.340822 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:52.340823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:52.340823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:52.340977 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:52.341050 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.341337 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.341406 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 04:05:52.341708 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.332101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:41291)
0927 04:05:52.332299 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.332345 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:52.332893 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:52.333172 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:52.333180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:52.333745 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:05:52.333936 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.333940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.334673 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.334676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.335562 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.335564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.335987 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.335989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.336112 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.336645 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:05:52.336653 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:05:52.338127 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:05:52.339972 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:52.339979 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:52.339981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:52.340296 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:52.340548 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:05:52.340549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:05:52.340549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:05:52.340722 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:05:52.341050 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:05:52.341055 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:05:52.341215 (+ 160us) client_negotiation.cc:769] Sending connection context
0927 04:05:52.341303 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 04:05:52.341428 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260927 04:05:52.342247 1514 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44953
I20260927 04:05:52.342343 1514 heartbeater.cc:464] Registering TS with master...
I20260927 04:05:52.342551 1514 heartbeater.cc:511] Master 127.17.184.254:44953 requested a full tablet report, sending...
I20260927 04:05:52.342978 1055 ts_manager.cc:194] Registered new tserver with Master: 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
I20260927 04:05:52.343387 1055 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:41291
I20260927 04:05:52.351918 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:05:52.359879 1104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.353248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39188 (local address 127.17.184.254:44953)
0927 04:05:52.353322 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.353324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.353416 (+ 92us) server_negotiation.cc:407] Connection header received
0927 04:05:52.353458 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.353459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.353469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.353523 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:05:52.354147 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.354774 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.356049 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.356252 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.356316 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.357520 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:05:52.357521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:05:52.357521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:05:52.357527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:05:52.358821 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:52.359189 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:52.359189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:52.359190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:52.359242 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:05:52.359397 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:05:52.359397 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:05:52.359397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:05:52.359536 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:05:52.359611 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.359728 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.359785 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 04:05:52.361570 1055 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:05:52.363255 1055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:05:52.369743 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.365840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.17.184.195:34793)
0927 04:05:52.366014 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.366020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.366031 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:05:52.366076 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.366079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.366095 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.366188 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:52.366378 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.367267 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.368736 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.369310 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.369400 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.369457 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.369617 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.369657 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260927 04:05:52.369733 1525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.365554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34793 (local address 127.0.0.1:44860)
0927 04:05:52.365847 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.365863 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:52.365924 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:52.366210 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:52.366213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:52.366215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:52.366288 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.366290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.367367 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.367368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.368651 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.368653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.369396 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.369397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.369484 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.369498 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 04:05:52.369598 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 04:05:52.369654 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":91,"threads_started":1}
I20260927 04:05:52.369884 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.365528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33023 (local address 127.0.0.1:54440)
0927 04:05:52.365699 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.365740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:52.366006 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:52.366241 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:52.366242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:52.366245 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:52.366321 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.366323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.367731 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.367732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.368765 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.368767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.369662 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.369663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.369729 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.369734 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:05:52.369764 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 04:05:52.369803 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":56,"threads_started":1}
I20260927 04:05:52.370074 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.365863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.17.184.193:35897)
0927 04:05:52.366111 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.366115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.366283 (+ 168us) server_negotiation.cc:407] Connection header received
0927 04:05:52.366326 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.366328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.366345 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.366487 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:52.366686 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.367776 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.368938 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.369583 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.369647 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.369710 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.369923 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.369975 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":78,"threads_started":1}
I20260927 04:05:52.370219 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.365725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:35897 (local address 127.0.0.1:49542)
0927 04:05:52.366146 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.366185 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:52.366231 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:52.366563 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:52.366564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:52.366565 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:52.366600 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.366601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.367885 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.367886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.368869 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.368872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.369740 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.369742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.369793 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.369796 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:05:52.369840 (+ 44us) client_negotiation.cc:240] Negotiation successful
0927 04:05:52.369871 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":74,"threads_started":1}
I20260927 04:05:52.370352 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.365672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54440 (local address 127.17.184.194:33023)
0927 04:05:52.366058 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.366062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.366072 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:05:52.366108 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.366110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.366126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.366400 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:52.366472 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.367628 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.368851 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.369555 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.370070 (+ 515us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.370134 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.370175 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.370219 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":73,"threads_started":1}
I20260927 04:05:52.370851 1448 tablet_service.cc:1467] Processing CreateTablet for tablet 85381bede498423db3f841a14174c5be (DEFAULT_TABLE table=test-table [id=57c94bc33c454cdfbe95f8d9be6e30f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:52.370873 1180 tablet_service.cc:1467] Processing CreateTablet for tablet 85381bede498423db3f841a14174c5be (DEFAULT_TABLE table=test-table [id=57c94bc33c454cdfbe95f8d9be6e30f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:52.371282 1180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85381bede498423db3f841a14174c5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:52.371282 1448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85381bede498423db3f841a14174c5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:52.371495 1314 tablet_service.cc:1467] Processing CreateTablet for tablet 85381bede498423db3f841a14174c5be (DEFAULT_TABLE table=test-table [id=57c94bc33c454cdfbe95f8d9be6e30f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:05:52.371784 1314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85381bede498423db3f841a14174c5be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:05:52.373935 1531 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Bootstrap starting.
I20260927 04:05:52.374632 1531 tablet_bootstrap.cc:654] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:52.374656 1532 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Bootstrap starting.
I20260927 04:05:52.375378 1532 tablet_bootstrap.cc:654] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:52.375484 1531 log.cc:824] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:52.375890 1533 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Bootstrap starting.
I20260927 04:05:52.375963 1532 log.cc:824] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:52.376454 1531 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: No bootstrap required, opened a new log
I20260927 04:05:52.376528 1533 tablet_bootstrap.cc:654] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Neither blocks nor log segments found. Creating new log.
I20260927 04:05:52.376547 1531 ts_tablet_manager.cc:1397] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 04:05:52.376714 1532 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: No bootstrap required, opened a new log
I20260927 04:05:52.376796 1532 ts_tablet_manager.cc:1397] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:05:52.377069 1533 log.cc:824] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Log is configured to *not* fsync() on all Append() calls
I20260927 04:05:52.377832 1533 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: No bootstrap required, opened a new log
I20260927 04:05:52.377902 1533 ts_tablet_manager.cc:1397] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:05:52.378310 1531 raft_consensus.cc:348] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.378480 1531 raft_consensus.cc:374] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:52.378526 1531 raft_consensus.cc:729] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d7a3620f864ca0bb7cd38b31c0f427, State: Initialized, Role: FOLLOWER
I20260927 04:05:52.378643 1531 consensus_queue.cc:260] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.378862 1532 raft_consensus.cc:348] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.379037 1531 ts_tablet_manager.cc:1428] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 04:05:52.379055 1532 raft_consensus.cc:374] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:52.379092 1532 raft_consensus.cc:729] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ab387c486444ebb27c57f6f83e578b, State: Initialized, Role: FOLLOWER
I20260927 04:05:52.379160 1246 heartbeater.cc:503] Master 127.17.184.254:44953 was elected leader, sending a full tablet report...
I20260927 04:05:52.379212 1532 consensus_queue.cc:260] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.379479 1514 heartbeater.cc:503] Master 127.17.184.254:44953 was elected leader, sending a full tablet report...
I20260927 04:05:52.379805 1532 ts_tablet_manager.cc:1428] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:05:52.380067 1533 raft_consensus.cc:348] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.380266 1533 raft_consensus.cc:374] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:05:52.380326 1533 raft_consensus.cc:729] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed9147adf4c432db5f4fcf65aab4a20, State: Initialized, Role: FOLLOWER
I20260927 04:05:52.380435 1533 consensus_queue.cc:260] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.380733 1533 ts_tablet_manager.cc:1428] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:05:52.380751 1380 heartbeater.cc:503] Master 127.17.184.254:44953 was elected leader, sending a full tablet report...
W20260927 04:05:52.434234 1381 tablet.cc:2369] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:05:52.514668 1247 tablet.cc:2369] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:05:52.542500 1537 raft_consensus.cc:482] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:05:52.542629 1537 raft_consensus.cc:504] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.542984 1537 leader_election.cc:288] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
I20260927 04:05:52.547319 1527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.543311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:55623 (local address 127.17.184.194:33023)
0927 04:05:52.543375 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.543378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.543393 (+ 15us) server_negotiation.cc:407] Connection header received
0927 04:05:52.543491 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.543492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.543501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.543608 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:52.543810 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.544699 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.546304 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.546928 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.547020 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.547030 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.547200 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.547247 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:05:52.547319 1529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.543365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:50301 (local address 127.17.184.195:34793)
0927 04:05:52.543432 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.543434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.543641 (+ 207us) server_negotiation.cc:407] Connection header received
0927 04:05:52.543673 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.543674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.543684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.543744 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:05:52.543928 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.544713 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.546305 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.546961 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.547027 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.547035 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.547200 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.547247 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:05:52.547338 1540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.543197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34793 (local address 127.17.184.193:50301)
0927 04:05:52.543464 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.543473 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:52.543530 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:52.543776 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:52.543778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:52.543779 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:52.543842 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.543844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.544796 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.544797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.546181 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.546185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.547030 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.547031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.547107 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.547114 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 04:05:52.547176 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 04:05:52.547220 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":125,"thread_start_us":50,"threads_started":1}
I20260927 04:05:52.547338 1541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.543197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33023 (local address 127.17.184.193:55623)
0927 04:05:52.543339 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.543351 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:05:52.543411 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:05:52.543628 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:05:52.543628 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:05:52.543630 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:05:52.543704 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.543706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.544798 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.544799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.546181 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:05:52.546185 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.547022 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.547023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:05:52.547107 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.547115 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 04:05:52.547183 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 04:05:52.547254 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":73,"thread_start_us":72,"threads_started":1}
I20260927 04:05:52.547832 1468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85381bede498423db3f841a14174c5be" candidate_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ab387c486444ebb27c57f6f83e578b" is_pre_election: true
I20260927 04:05:52.548000 1468 raft_consensus.cc:2454] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d7a3620f864ca0bb7cd38b31c0f427 in term 0.
I20260927 04:05:52.548007 1334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85381bede498423db3f841a14174c5be" candidate_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" is_pre_election: true
I20260927 04:05:52.548178 1334 raft_consensus.cc:2454] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d7a3620f864ca0bb7cd38b31c0f427 in term 0.
I20260927 04:05:52.548273 1133 leader_election.cc:302] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427, 12ab387c486444ebb27c57f6f83e578b; no voters:
I20260927 04:05:52.548496 1537 raft_consensus.cc:2781] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:05:52.548601 1537 raft_consensus.cc:482] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:05:52.548627 1537 raft_consensus.cc:3037] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:52.549391 1537 raft_consensus.cc:504] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.549539 1537 leader_election.cc:288] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
I20260927 04:05:52.549693 1468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85381bede498423db3f841a14174c5be" candidate_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ab387c486444ebb27c57f6f83e578b"
I20260927 04:05:52.549710 1334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85381bede498423db3f841a14174c5be" candidate_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed9147adf4c432db5f4fcf65aab4a20"
I20260927 04:05:52.549778 1468 raft_consensus.cc:3037] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:52.549778 1334 raft_consensus.cc:3037] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:05:52.550657 1334 raft_consensus.cc:2454] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d7a3620f864ca0bb7cd38b31c0f427 in term 1.
I20260927 04:05:52.550668 1468 raft_consensus.cc:2454] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d7a3620f864ca0bb7cd38b31c0f427 in term 1.
I20260927 04:05:52.550860 1135 leader_election.cc:302] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427, 0ed9147adf4c432db5f4fcf65aab4a20; no voters:
I20260927 04:05:52.550973 1537 raft_consensus.cc:2781] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:05:52.551179 1537 raft_consensus.cc:686] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 LEADER]: Becoming Leader. State: Replica: 02d7a3620f864ca0bb7cd38b31c0f427, State: Running, Role: LEADER
I20260927 04:05:52.551277 1537 consensus_queue.cc:237] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:05:52.552119 1054 catalog_manager.cc:5495] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d7a3620f864ca0bb7cd38b31c0f427 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } health_report { overall_health: UNKNOWN } } }
W20260927 04:05:52.582063 1515 tablet.cc:2369] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:05:52.598531 1530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.595977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49546 (local address 127.17.184.193:35897)
0927 04:05:52.596041 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0927 04:05:52.596043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:05:52.596054 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:05:52.596105 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:05:52.596106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:05:52.596126 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:05:52.596194 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:05:52.596332 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.596792 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.597536 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:05:52.597741 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:05:52.597803 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:05:52.598003 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:05:52.598146 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:05:52.598231 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:05:52.598389 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 04:05:52.598421 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 04:05:52.600558 1468 raft_consensus.cc:1261] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 1 FOLLOWER]: Refusing update from remote peer 02d7a3620f864ca0bb7cd38b31c0f427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:52.600555 1334 raft_consensus.cc:1261] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 1 FOLLOWER]: Refusing update from remote peer 02d7a3620f864ca0bb7cd38b31c0f427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:05:52.601073 1542 consensus_queue.cc:1035] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:52.601224 1537 consensus_queue.cc:1035] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:05:52.603749 1545 mvcc.cc:205] Tried to move back new op lower bound from 7333814077849055232 to 7333814077650702336. Current Snapshot: MvccSnapshot[applied={T|T < 7333814077849055232}]
I20260927 04:05:52.604369 1546 mvcc.cc:205] Tried to move back new op lower bound from 7333814077849055232 to 7333814077650702336. Current Snapshot: MvccSnapshot[applied={T|T < 7333814077849055232}]
I20260927 04:05:52.605372 1547 mvcc.cc:205] Tried to move back new op lower bound from 7333814077849055232 to 7333814077650702336. Current Snapshot: MvccSnapshot[applied={T|T < 7333814077849055232}]
I20260927 04:05:52.606237 1055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:05:52.606339 1055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:05:52.607126 1055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Sending DeleteTablet for 3 replicas of tablet 85381bede498423db3f841a14174c5be
I20260927 04:05:52.607506 1314 tablet_service.cc:1514] Processing DeleteTablet for tablet 85381bede498423db3f841a14174c5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:52 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:54440
I20260927 04:05:52.607539 1180 tablet_service.cc:1514] Processing DeleteTablet for tablet 85381bede498423db3f841a14174c5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:52 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:49542
I20260927 04:05:52.607781 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 1117
I20260927 04:05:52.607800 1554 tablet_replica.cc:330] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: stopping tablet replica
I20260927 04:05:52.607909 1554 raft_consensus.cc:2229] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:05:52.609700 1448 tablet_service.cc:1514] Processing DeleteTablet for tablet 85381bede498423db3f841a14174c5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:52 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:44860
I20260927 04:05:52.609942 1556 tablet_replica.cc:330] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: stopping tablet replica
I20260927 04:05:52.610062 1556 raft_consensus.cc:2229] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:05:52.610199 1556 raft_consensus.cc:2258] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:52.610658 1556 ts_tablet_manager.cc:1905] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:52.611218 1555 tablet_replica.cc:330] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: stopping tablet replica
I20260927 04:05:52.611371 1555 raft_consensus.cc:2229] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:05:52.611517 1555 raft_consensus.cc:2258] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:05:52.611989 1555 ts_tablet_manager.cc:1905] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:05:52.612551 1556 ts_tablet_manager.cc:1918] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:05:52.612627 1556 log.cc:1196] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/85381bede498423db3f841a14174c5be
I20260927 04:05:52.612917 1556 ts_tablet_manager.cc:1939] T 85381bede498423db3f841a14174c5be P 12ab387c486444ebb27c57f6f83e578b: Deleting consensus metadata
I20260927 04:05:52.613374 1039 catalog_manager.cc:4841] TS 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793): tablet 85381bede498423db3f841a14174c5be (table test-table [id=57c94bc33c454cdfbe95f8d9be6e30f9]) successfully deleted
W20260927 04:05:52.613540 1041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:35897 (error 108)
I20260927 04:05:52.613739 1555 ts_tablet_manager.cc:1918] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:05:52.613834 1555 log.cc:1196] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/85381bede498423db3f841a14174c5be
I20260927 04:05:52.614043 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.613910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:35897 (local address 127.0.0.1:49558)
0927 04:05:52.613965 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.614002 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:35897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 04:05:52.614125 1555 ts_tablet_manager.cc:1939] T 85381bede498423db3f841a14174c5be P 0ed9147adf4c432db5f4fcf65aab4a20: Deleting consensus metadata
W20260927 04:05:52.614161 1041 catalog_manager.cc:4584] TS 02d7a3620f864ca0bb7cd38b31c0f427 (127.17.184.193:35897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85381bede498423db3f841a14174c5be: Network error: Client connection negotiation failed: client connection to 127.17.184.193:35897: connect: Connection refused (error 111)
I20260927 04:05:52.614272 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 1251
I20260927 04:05:52.619740 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 1385
I20260927 04:05:52.619791 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:05:52.619630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33023 (local address 127.0.0.1:54454)
0927 04:05:52.619690 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 04:05:52.619727 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.194:33023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:05:52.624101 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 1023
I20260927 04:06:12.629598 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44953
--webserver_interface=127.17.184.254
--webserver_port=33353
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:12.712271 1558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:12.712478 1558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:12.712502 1558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:12.713711 1558 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 04:06:12.713755 1558 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 04:06:12.713769 1558 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 04:06:12.713780 1558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:12.713825 1558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:12.713852 1558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:12.713869 1558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:12.713892 1558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:12.715489 1558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=33353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:12.715760 1558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:12.716013 1558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:12.725273 1564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:12.725327 1565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:12.725414 1558 server_base.cc:1034] running on GCE node
W20260927 04:06:12.725273 1567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:12.725977 1558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:12.726279 1558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:12.727430 1558 hybrid_clock.cc:648] HybridClock initialized: now 1790481972727417 us; error 30 us; skew 500 ppm
Sep 27 04:06:12 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481972, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:12.738957 1558 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:06:12.739563 1558 webserver.cc:460] Webserver started at http://127.17.184.254:33353/ using document root <none> and password file <none>
I20260927 04:06:12.739846 1558 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:12.739941 1558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:12.741477 1558 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:12.742383 1574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:12.742686 1558 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:12.742808 1558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "a0233c61cf5e3c43f625c488551bcd86"
server_key_iv: "2f44e17c050daf890f7746783147ac49"
server_key_version: "encryptionkey@0"
I20260927 04:06:12.743304 1558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:12.751928 1558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:12.754390 1558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:12.754623 1558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:12.759328 1558 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44953
I20260927 04:06:12.759380 1626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44953 every 8 connection(s)
I20260927 04:06:12.759837 1558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:12.761371 1627 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:06:12.762051 1627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Bootstrap starting.
I20260927 04:06:12.765609 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 1558
I20260927 04:06:12.765990 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:35897
--local_ip_for_outbound_sockets=127.17.184.193
--tserver_master_addrs=127.17.184.254:44953
--webserver_port=40785
--webserver_interface=127.17.184.193
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:06:12.771250 1627 log.cc:824] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:12.773258 1627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:06:12.773589 1627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Bootstrap complete.
I20260927 04:06:12.775967 1627 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:06:12.776193 1627 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff842ae9f3b4bd2a33cf37bd61d6a8d, State: Initialized, Role: FOLLOWER
I20260927 04:06:12.776361 1627 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:06:12.776437 1627 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:12.776490 1627 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:12.776539 1627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 1 FOLLOWER]: Advancing to term 2
I20260927 04:06:12.777585 1627 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:06:12.777705 1627 leader_election.cc:302] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff842ae9f3b4bd2a33cf37bd61d6a8d; no voters:
I20260927 04:06:12.777916 1627 leader_election.cc:288] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 04:06:12.777953 1631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 04:06:12.778064 1631 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 2 LEADER]: Becoming Leader. State: Replica: dff842ae9f3b4bd2a33cf37bd61d6a8d, State: Running, Role: LEADER
I20260927 04:06:12.778199 1627 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:12.778221 1631 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:06:12.778584 1633 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff842ae9f3b4bd2a33cf37bd61d6a8d. Latest consensus state: current_term: 2 leader_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } } }
I20260927 04:06:12.778659 1633 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:12.778818 1632 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } } }
I20260927 04:06:12.778914 1632 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:12.779173 1637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:12.780020 1637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:06:12.780181 1637 catalog_manager.cc:1259] Loaded cluster ID: d76cfbdc2ba4425c80d8a804c91427ce
I20260927 04:06:12.780381 1637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:06:12.781522 1637 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:12.781685 1637 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Loaded TSK: 0
I20260927 04:06:12.795042 1637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Generated new TSK 1
I20260927 04:06:12.795159 1637 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 04:06:12.854885 1629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:12.855213 1629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:12.855254 1629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:12.856503 1629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:12.856578 1629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:12.856624 1629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:06:12.858117 1629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:35897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=40785
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:12.858379 1629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:12.858613 1629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:12.864894 1655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:12.864898 1656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:12.864933 1658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:12.865177 1629 server_base.cc:1034] running on GCE node
I20260927 04:06:12.865393 1629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:12.865666 1629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:12.866827 1629 hybrid_clock.cc:648] HybridClock initialized: now 1790481972866803 us; error 40 us; skew 500 ppm
Sep 27 04:06:12 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481972, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:12.874694 1629 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:06:12.875325 1629 webserver.cc:460] Webserver started at http://127.17.184.193:40785/ using document root <none> and password file <none>
I20260927 04:06:12.875578 1629 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:12.875664 1629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:12.877163 1629 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:12.877907 1665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:12.878113 1629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:12.878273 1629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "02d7a3620f864ca0bb7cd38b31c0f427"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "9f9f72dd24e49bc50d154f85377ff794"
server_key_iv: "c8d44fe05389f355ca004c5feead5f72"
server_key_version: "encryptionkey@0"
I20260927 04:06:12.878754 1629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:12.889140 1629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:12.892179 1629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:12.892365 1629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:12.892663 1629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:12.893201 1672 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:06:12.894851 1629 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:06:12.894939 1629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260927 04:06:12.894979 1629 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:06:12.895871 1629 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:06:12.895921 1629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:06:12.896040 1672 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Bootstrap starting.
I20260927 04:06:12.903604 1629 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:35897
I20260927 04:06:12.904135 1629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:06:12.907150 1779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:35897 every 8 connection(s)
I20260927 04:06:12.907344 1672 log.cc:824] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:12.910468 1672 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:06:12.910816 1672 tablet_bootstrap.cc:492] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Bootstrap complete.
I20260927 04:06:12.911836 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 1629
I20260927 04:06:12.912034 1672 ts_tablet_manager.cc:1397] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20260927 04:06:12.912158 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:33023
--local_ip_for_outbound_sockets=127.17.184.194
--tserver_master_addrs=127.17.184.254:44953
--webserver_port=39155
--webserver_interface=127.17.184.194
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:06:12.914095 1672 raft_consensus.cc:348] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:06:12.914520 1672 raft_consensus.cc:729] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d7a3620f864ca0bb7cd38b31c0f427, State: Initialized, Role: FOLLOWER
I20260927 04:06:12.915226 1672 consensus_queue.cc:260] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } }
I20260927 04:06:12.915645 1672 ts_tablet_manager.cc:1428] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
Sep 27 04:06:12 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481972, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:12.920936 1783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:12.907848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:59409)
0927 04:06:12.908157 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:12.908206 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:12.908957 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:12.909510 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:12.909514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:12.909933 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:12.910236 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:12.910243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:12.911979 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:12.911980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:12.913351 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:12.913354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:12.914165 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:12.914167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:12.914317 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:12.915011 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:12.915023 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:12.916760 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:12.919058 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:12.919066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:12.919068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:12.919353 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:12.919684 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:12.919685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:12.919685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:12.919802 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:12.920256 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:12.920263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:12.920401 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 04:06:12.920510 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 04:06:12.920716 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":102,"threads_started":1}
I20260927 04:06:12.920955 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:12.907921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:59409 (local address 127.17.184.254:44953)
0927 04:06:12.908103 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:12.908113 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:12.908343 (+ 230us) server_negotiation.cc:407] Connection header received
0927 04:06:12.909221 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:12.909229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:12.909291 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:12.909453 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:12.910396 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:12.911836 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:12.913462 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:12.914056 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:12.914191 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:12.916875 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:12.916880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:12.916880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:12.916893 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:12.918942 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:12.919482 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:12.919486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:12.919486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:12.919604 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:12.919879 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:12.919880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:12.919880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:12.920196 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:12.920280 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:12.920580 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 04:06:12.920789 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260927 04:06:12.921751 1780 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44953
I20260927 04:06:12.921873 1780 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:12.922088 1780 heartbeater.cc:511] Master 127.17.184.254:44953 requested a full tablet report, sending...
I20260927 04:06:12.922971 1591 ts_manager.cc:194] Registered new tserver with Master: 02d7a3620f864ca0bb7cd38b31c0f427 (127.17.184.193:35897)
I20260927 04:06:12.924160 1591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:59409
I20260927 04:06:12.925235 1780 heartbeater.cc:503] Master 127.17.184.254:44953 was elected leader, sending a full tablet report...
Sep 27 04:06:12 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481972, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.193@KRBTEST.COM
I20260927 04:06:12.932772 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:12.923577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:35897 (local address 127.0.0.1:55308)
0927 04:06:12.924020 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:12.924056 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:12.924122 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:12.924441 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:12.924446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:12.924817 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:12.924906 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:12.924908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:12.926050 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:12.926051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:12.927026 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:12.927029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:12.927542 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:12.927543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:12.927637 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:12.928177 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:12.928181 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:12.929522 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:12.931503 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:12.931511 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:12.931513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:12.931733 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:12.931969 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:12.931970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:12.931970 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:12.932035 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:12.932360 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:12.932366 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:12.932421 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 04:06:12.932528 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 04:06:12.932664 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":131,"threads_started":1}
I20260927 04:06:12.933269 1788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:12.923867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55308 (local address 127.17.184.193:35897)
0927 04:06:12.924253 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:12.924258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:12.924268 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:06:12.924302 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:12.924304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:12.924317 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:12.924377 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:12.924995 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:12.925939 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:12.927133 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:12.927444 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:12.927538 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:12.929627 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:12.929641 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:12.929642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:12.929648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:12.931396 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:12.931833 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:12.931836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:12.931837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:12.931899 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:12.932115 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:12.932116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:12.932117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:12.932283 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:12.932380 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:12.932565 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 04:06:12.932664 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":75,"threads_started":1}
I20260927 04:06:12.933617 1714 tablet_service.cc:1514] Processing DeleteTablet for tablet 85381bede498423db3f841a14174c5be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:05:52 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:55308
I20260927 04:06:12.934139 1789 tablet_replica.cc:330] stopping tablet replica
I20260927 04:06:12.934274 1789 raft_consensus.cc:2229] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:12.934350 1789 raft_consensus.cc:2258] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:12.934969 1789 ts_tablet_manager.cc:1905] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:12.937520 1789 ts_tablet_manager.cc:1918] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:12.937634 1789 log.cc:1196] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal/wals/85381bede498423db3f841a14174c5be
I20260927 04:06:12.937912 1789 ts_tablet_manager.cc:1939] T 85381bede498423db3f841a14174c5be P 02d7a3620f864ca0bb7cd38b31c0f427: Deleting consensus metadata
I20260927 04:06:12.938459 1577 catalog_manager.cc:4841] TS 02d7a3620f864ca0bb7cd38b31c0f427 (127.17.184.193:35897): tablet 85381bede498423db3f841a14174c5be (table test-table [id=57c94bc33c454cdfbe95f8d9be6e30f9]) successfully deleted
W20260927 04:06:13.025564 1785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:13.025825 1785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:13.025857 1785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:13.027431 1785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:13.027488 1785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:13.027524 1785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:06:13.029291 1785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=39155
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:13.029585 1785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:13.029867 1785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:13.035902 1794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:13.035902 1795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:13.036499 1797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:13.036689 1785 server_base.cc:1034] running on GCE node
I20260927 04:06:13.036983 1785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:13.037322 1785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:13.038537 1785 hybrid_clock.cc:648] HybridClock initialized: now 1790481973038518 us; error 53 us; skew 500 ppm
Sep 27 04:06:13 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481973, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:13.041558 1785 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:06:13.042179 1785 webserver.cc:460] Webserver started at http://127.17.184.194:39155/ using document root <none> and password file <none>
I20260927 04:06:13.042383 1785 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:13.042435 1785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:13.044490 1785 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:13.045389 1804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:13.045668 1785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:13.045801 1785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "0ed9147adf4c432db5f4fcf65aab4a20"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "73092d704412955dd5469ebc08935a74"
server_key_iv: "c3efeb6ea77d5391485e4748c8427e76"
server_key_version: "encryptionkey@0"
I20260927 04:06:13.046334 1785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:13.070391 1785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:13.073244 1785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:13.073379 1785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:13.073606 1785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:13.073982 1785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:13.074023 1785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:13.074050 1785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:13.074066 1785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:13.081302 1785 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33023
I20260927 04:06:13.081343 1917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33023 every 8 connection(s)
I20260927 04:06:13.081755 1785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:06:13.082760 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 1785
I20260927 04:06:13.083141 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:34793
--local_ip_for_outbound_sockets=127.17.184.195
--tserver_master_addrs=127.17.184.254:44953
--webserver_port=35745
--webserver_interface=127.17.184.195
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 04:06:13 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481973, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:13.093940 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.082559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:35809 (local address 127.17.184.254:44953)
0927 04:06:13.082609 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.082611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.082994 (+ 383us) server_negotiation.cc:407] Connection header received
0927 04:06:13.083693 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.083694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.083713 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.083801 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:13.084853 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.085772 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.087452 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.087712 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.087782 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.090582 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:13.090584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:13.090584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:13.090593 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:13.092296 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:13.092812 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:13.092813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:13.092814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:13.092881 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:13.093175 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:13.093177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:13.093177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:13.093310 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:13.093393 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.093725 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.093832 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:06:13.094146 1920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.082625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:35809)
0927 04:06:13.082816 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:13.082868 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:13.083567 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:13.083886 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:13.083890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:13.084538 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:13.084740 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.084744 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.085907 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.085908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.087311 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.087315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.087827 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.087828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.087999 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.088616 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:13.088629 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:13.090455 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:13.092435 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:13.092445 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:13.092447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:13.092710 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:13.092964 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:13.092965 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:13.092965 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:13.093099 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:13.093397 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:13.093404 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:13.093565 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 04:06:13.093688 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 04:06:13.093871 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260927 04:06:13.094683 1918 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44953
I20260927 04:06:13.094805 1918 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:13.095043 1918 heartbeater.cc:511] Master 127.17.184.254:44953 requested a full tablet report, sending...
I20260927 04:06:13.095505 1591 ts_manager.cc:194] Registered new tserver with Master: 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023)
I20260927 04:06:13.095947 1591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:35809
W20260927 04:06:13.166728 1921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:13.166976 1921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:13.167003 1921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:13.168226 1921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:13.168270 1921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:13.168298 1921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:06:13.169804 1921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:34793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=35745
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:13.170063 1921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:13.170342 1921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:13.175490 1929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:13.175489 1927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:13.175494 1926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:13.175828 1921 server_base.cc:1034] running on GCE node
I20260927 04:06:13.176062 1921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:13.176359 1921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:13.177525 1921 hybrid_clock.cc:648] HybridClock initialized: now 1790481973177522 us; error 47 us; skew 500 ppm
Sep 27 04:06:13 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481973, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:13.180365 1921 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:06:13.180791 1921 webserver.cc:460] Webserver started at http://127.17.184.195:35745/ using document root <none> and password file <none>
I20260927 04:06:13.181022 1921 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:13.181080 1921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:13.182492 1921 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:13.183244 1936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:13.183449 1921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:13.183554 1921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "12ab387c486444ebb27c57f6f83e578b"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "5e6038173d98851baea9e666b8bf895b"
server_key_iv: "c09bfa6eacee44daf37df46878976c19"
server_key_version: "encryptionkey@0"
I20260927 04:06:13.183982 1921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:13.194063 1921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:13.196470 1921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:13.196638 1921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:13.196882 1921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:13.197207 1921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:13.197264 1921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:13.197304 1921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:13.197332 1921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:13.203284 1921 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:34793
I20260927 04:06:13.203365 2049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:34793 every 8 connection(s)
I20260927 04:06:13.203739 1921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:06:13.209558 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 1921
Sep 27 04:06:13 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481973, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:13.214744 1784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.204522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:57737 (local address 127.17.184.254:44953)
0927 04:06:13.204607 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.204610 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.204829 (+ 219us) server_negotiation.cc:407] Connection header received
0927 04:06:13.205430 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.205431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.205444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.205511 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:13.206595 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.207244 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.208370 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.208598 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.208699 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.211382 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:13.211384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:13.211384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:13.211391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:13.213066 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:13.213538 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:13.213540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:13.213540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:13.213704 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:13.214045 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:13.214046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:13.214047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:13.214204 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:13.214287 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.214599 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.214676 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 04:06:13.215142 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.204485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:57737)
0927 04:06:13.204674 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:13.204731 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:13.205312 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:13.205599 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:13.205603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:13.206258 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:13.206476 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.206481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.207370 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.207371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.208265 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.208267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.208709 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.208710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.208871 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.209377 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:13.209387 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:13.211270 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:13.213139 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:13.213145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:13.213148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:13.213463 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:13.213842 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:13.213843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:13.213844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:13.213965 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:13.214305 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:13.214311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:13.214436 (+ 125us) client_negotiation.cc:769] Sending connection context
0927 04:06:13.214544 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:06:13.214832 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260927 04:06:13.215734 2050 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44953
I20260927 04:06:13.215852 2050 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:13.216044 2050 heartbeater.cc:511] Master 127.17.184.254:44953 requested a full tablet report, sending...
I20260927 04:06:13.216428 1591 ts_manager.cc:194] Registered new tserver with Master: 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
I20260927 04:06:13.216835 1591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:57737
I20260927 04:06:13.217986 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.210007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.17.184.254:44953)
0927 04:06:13.210120 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.210121 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.210246 (+ 125us) server_negotiation.cc:407] Connection header received
0927 04:06:13.210310 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.210311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.210325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.210393 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:13.211094 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.211755 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.212974 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.213256 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.213329 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.214619 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:13.214621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:13.214621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:13.214628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:13.216308 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:13.216675 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:13.216676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:13.216676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:13.216741 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:13.216939 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:13.216940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:13.216941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:13.217048 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:13.217231 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.217808 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.217885 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260927 04:06:13.218722 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:06:13.221321 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.219100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.17.184.254:44953)
0927 04:06:13.219175 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.219177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.219215 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:06:13.219292 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.219293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.219321 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.219377 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:06:13.219455 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.219908 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.220657 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.220877 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.220940 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.221071 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:06:13.221118 (+ 47us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 04:06:13.221275 (+ 157us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51036: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 04:06:13.221405 2054 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51036: token expired
W20260927 04:06:13.221441 1519 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.184.254:44953)
I20260927 04:06:13.228096 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.221687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.17.184.254:44953)
0927 04:06:13.221738 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.221740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.221749 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:13.221794 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.221795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.221802 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.221847 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:13.222414 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.222995 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.223671 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.223841 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.223895 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.224973 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:13.224975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:13.224975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:13.224982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:13.226404 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:13.226990 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:13.226992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:13.226992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:13.227063 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:13.227358 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:13.227360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:13.227360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:13.227525 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:13.227634 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.227807 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.227872 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 04:06:13.229338 1591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:06:13.231040 1591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:06:13.233274 1714 tablet_service.cc:1467] Processing CreateTablet for tablet fe4cc445dc944c52b2f70d287b79fa3b (DEFAULT_TABLE table=test-table [id=a66640af02b64315bf06f7f59d1bff11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:13.233489 1714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4cc445dc944c52b2f70d287b79fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:13.235225 1672 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Bootstrap starting.
I20260927 04:06:13.235536 1672 tablet_bootstrap.cc:654] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:13.236490 1672 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: No bootstrap required, opened a new log
I20260927 04:06:13.236544 1672 ts_tablet_manager.cc:1397] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260927 04:06:13.236692 1672 raft_consensus.cc:348] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.236758 1787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.232933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33023 (local address 127.0.0.1:56366)
0927 04:06:13.233019 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:13.233031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:13.233073 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:13.233553 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:13.233554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:13.233555 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:13.233607 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.233609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.234599 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.234601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.235634 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.235637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.236480 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.236481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.236555 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.236578 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 04:06:13.236624 (+ 46us) client_negotiation.cc:240] Negotiation successful
0927 04:06:13.236682 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 04:06:13.236768 1672 raft_consensus.cc:374] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:13.236812 1672 raft_consensus.cc:729] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d7a3620f864ca0bb7cd38b31c0f427, State: Initialized, Role: FOLLOWER
I20260927 04:06:13.236860 1672 consensus_queue.cc:260] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.236944 1672 ts_tablet_manager.cc:1428] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:13.237174 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.233183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.17.184.194:33023)
0927 04:06:13.233360 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.233364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.233373 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:13.233437 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.233440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.233456 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.233535 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:13.233667 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.234509 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.235723 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.236354 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.236484 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.236668 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.236699 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.236740 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":111,"threads_started":1}
I20260927 04:06:13.237540 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.232973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34793 (local address 127.0.0.1:38592)
0927 04:06:13.233142 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:13.233153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:13.233203 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:13.233846 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:13.233847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:13.233849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:13.233883 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.233885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.235153 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.235154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.236254 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.236256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.237273 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.237274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.237343 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.237346 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:13.237397 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:06:13.237452 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260927 04:06:13.237768 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.233068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38592 (local address 127.17.184.195:34793)
0927 04:06:13.233460 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.233464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.233476 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:06:13.233741 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.233744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.233763 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.233852 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:13.233994 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.235050 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.236360 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.237119 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.237185 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.237241 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.237642 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.237687 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":72,"threads_started":1}
I20260927 04:06:13.237926 1852 tablet_service.cc:1467] Processing CreateTablet for tablet fe4cc445dc944c52b2f70d287b79fa3b (DEFAULT_TABLE table=test-table [id=a66640af02b64315bf06f7f59d1bff11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:13.238284 1852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4cc445dc944c52b2f70d287b79fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:13.238420 1984 tablet_service.cc:1467] Processing CreateTablet for tablet fe4cc445dc944c52b2f70d287b79fa3b (DEFAULT_TABLE table=test-table [id=a66640af02b64315bf06f7f59d1bff11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:13.238649 1984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4cc445dc944c52b2f70d287b79fa3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:13.241077 2060 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: Bootstrap starting.
I20260927 04:06:13.241341 2061 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: Bootstrap starting.
I20260927 04:06:13.241780 2060 tablet_bootstrap.cc:654] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:13.242043 2061 tablet_bootstrap.cc:654] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:13.242447 2060 log.cc:824] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:13.242632 2061 log.cc:824] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:13.243426 2060 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: No bootstrap required, opened a new log
I20260927 04:06:13.243438 2061 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: No bootstrap required, opened a new log
I20260927 04:06:13.243505 2060 ts_tablet_manager.cc:1397] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:13.243516 2061 ts_tablet_manager.cc:1397] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:13.245633 2061 raft_consensus.cc:348] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.245587 2060 raft_consensus.cc:348] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.245787 2061 raft_consensus.cc:374] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:13.245806 2060 raft_consensus.cc:374] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:13.245826 2061 raft_consensus.cc:729] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12ab387c486444ebb27c57f6f83e578b, State: Initialized, Role: FOLLOWER
I20260927 04:06:13.245841 2060 raft_consensus.cc:729] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed9147adf4c432db5f4fcf65aab4a20, State: Initialized, Role: FOLLOWER
I20260927 04:06:13.245946 2061 consensus_queue.cc:260] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.245946 2060 consensus_queue.cc:260] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.246243 2061 ts_tablet_manager.cc:1428] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:06:13.246243 2060 ts_tablet_manager.cc:1428] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:06:13.246294 1918 heartbeater.cc:503] Master 127.17.184.254:44953 was elected leader, sending a full tablet report...
I20260927 04:06:13.246371 2050 heartbeater.cc:503] Master 127.17.184.254:44953 was elected leader, sending a full tablet report...
I20260927 04:06:13.267818 1786 raft_consensus.cc:482] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:13.267951 1786 raft_consensus.cc:504] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.268379 1786 leader_election.cc:288] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
I20260927 04:06:13.271324 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.268634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:35525 (local address 127.17.184.194:33023)
0927 04:06:13.268727 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.268729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.268742 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:06:13.268772 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.268773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.268785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.268863 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:13.268998 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.269509 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.270515 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.270937 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.271031 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.271041 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.271217 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.271257 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 04:06:13.271324 1783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.268570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33023 (local address 127.17.184.193:35525)
0927 04:06:13.268627 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:13.268639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:13.268704 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:13.268865 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:13.268866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:13.268867 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:13.268926 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.268932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.269598 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.269599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.270423 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.270426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.271031 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.271032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.271107 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.271128 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 04:06:13.271207 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 04:06:13.271252 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 04:06:13.271391 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.268656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:44207 (local address 127.17.184.195:34793)
0927 04:06:13.268710 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.268713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.268857 (+ 144us) server_negotiation.cc:407] Connection header received
0927 04:06:13.268945 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.268946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.268955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.269048 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:13.269196 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.269657 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.270590 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.271018 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.271115 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.271125 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.271288 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.271324 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:06:13.271629 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.268588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34793 (local address 127.17.184.193:44207)
0927 04:06:13.268786 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:13.268794 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:13.268872 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:13.269041 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:13.269042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:13.269044 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:13.269128 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.269130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.269741 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.269742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.270491 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:13.270493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.271132 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.271133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:13.271204 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.271208 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:13.271270 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 04:06:13.271311 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":117,"threads_started":1}
I20260927 04:06:13.271713 1872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cc445dc944c52b2f70d287b79fa3b" candidate_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" is_pre_election: true
I20260927 04:06:13.271909 1872 raft_consensus.cc:2454] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d7a3620f864ca0bb7cd38b31c0f427 in term 0.
I20260927 04:06:13.272200 2004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cc445dc944c52b2f70d287b79fa3b" candidate_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ab387c486444ebb27c57f6f83e578b" is_pre_election: true
I20260927 04:06:13.272363 2004 raft_consensus.cc:2454] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d7a3620f864ca0bb7cd38b31c0f427 in term 0.
I20260927 04:06:13.272513 1668 leader_election.cc:302] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427, 0ed9147adf4c432db5f4fcf65aab4a20; no voters:
I20260927 04:06:13.272688 1786 raft_consensus.cc:2781] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:06:13.272745 1786 raft_consensus.cc:482] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:06:13.272761 1786 raft_consensus.cc:3037] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:13.273500 1786 raft_consensus.cc:504] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.273651 1786 leader_election.cc:288] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 1 election: Requested vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
I20260927 04:06:13.273808 1872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cc445dc944c52b2f70d287b79fa3b" candidate_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ed9147adf4c432db5f4fcf65aab4a20"
I20260927 04:06:13.273856 2004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4cc445dc944c52b2f70d287b79fa3b" candidate_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12ab387c486444ebb27c57f6f83e578b"
I20260927 04:06:13.273891 1872 raft_consensus.cc:3037] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:13.273962 2004 raft_consensus.cc:3037] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:13.274699 1872 raft_consensus.cc:2454] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d7a3620f864ca0bb7cd38b31c0f427 in term 1.
I20260927 04:06:13.274760 2004 raft_consensus.cc:2454] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d7a3620f864ca0bb7cd38b31c0f427 in term 1.
I20260927 04:06:13.274890 1668 leader_election.cc:302] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427, 0ed9147adf4c432db5f4fcf65aab4a20; no voters:
I20260927 04:06:13.275033 1786 raft_consensus.cc:2781] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:13.275245 1786 raft_consensus.cc:686] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 LEADER]: Becoming Leader. State: Replica: 02d7a3620f864ca0bb7cd38b31c0f427, State: Running, Role: LEADER
I20260927 04:06:13.275348 1786 consensus_queue.cc:237] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:13.276136 1591 catalog_manager.cc:5495] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d7a3620f864ca0bb7cd38b31c0f427 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } health_report { overall_health: HEALTHY } } }
I20260927 04:06:13.286145 1788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.283773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.17.184.193:35897)
0927 04:06:13.283812 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:13.283814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:13.283822 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:06:13.283865 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:13.283866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:13.283888 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:13.283965 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:06:13.284074 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.284582 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.285322 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:13.285529 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:13.285605 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:13.285779 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:06:13.285875 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:06:13.285959 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:13.286000 (+ 41us) server_negotiation.cc:299] Negotiation successful
0927 04:06:13.286036 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 04:06:13.287946 2004 raft_consensus.cc:1261] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 1 FOLLOWER]: Refusing update from remote peer 02d7a3620f864ca0bb7cd38b31c0f427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:13.287978 1872 raft_consensus.cc:1261] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 1 FOLLOWER]: Refusing update from remote peer 02d7a3620f864ca0bb7cd38b31c0f427: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:13.288341 2067 consensus_queue.cc:1035] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:13.288496 1786 consensus_queue.cc:1035] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:13.291052 2069 mvcc.cc:205] Tried to move back new op lower bound from 7333814162584813568 to 7333814162536394752. Current Snapshot: MvccSnapshot[applied={T|T < 7333814162584813568}]
I20260927 04:06:13.292212 2071 mvcc.cc:205] Tried to move back new op lower bound from 7333814162584813568 to 7333814162536394752. Current Snapshot: MvccSnapshot[applied={T|T < 7333814162584813568 or (T in {7333814162584813568})}]
I20260927 04:06:13.292410 2070 mvcc.cc:205] Tried to move back new op lower bound from 7333814162584813568 to 7333814162536394752. Current Snapshot: MvccSnapshot[applied={T|T < 7333814162584813568 or (T in {7333814162584813568})}]
I20260927 04:06:13.293668 1590 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:13.293785 1590 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:13.294709 1590 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Sending DeleteTablet for 3 replicas of tablet fe4cc445dc944c52b2f70d287b79fa3b
I20260927 04:06:13.295132 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 1629
I20260927 04:06:13.295143 1852 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4cc445dc944c52b2f70d287b79fa3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:13 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:56366
I20260927 04:06:13.295145 1984 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4cc445dc944c52b2f70d287b79fa3b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:13 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:38592
I20260927 04:06:13.296087 2078 tablet_replica.cc:330] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: stopping tablet replica
I20260927 04:06:13.296200 2078 raft_consensus.cc:2229] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:13.296305 2078 raft_consensus.cc:2258] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:13.296479 2079 tablet_replica.cc:330] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: stopping tablet replica
I20260927 04:06:13.296607 2079 raft_consensus.cc:2229] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:13.296715 2079 raft_consensus.cc:2258] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:13.296840 2078 ts_tablet_manager.cc:1905] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:13.297150 2079 ts_tablet_manager.cc:1905] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:13.298998 2079 ts_tablet_manager.cc:1918] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:13.298998 2078 ts_tablet_manager.cc:1918] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:13.299078 2079 log.cc:1196] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/fe4cc445dc944c52b2f70d287b79fa3b
I20260927 04:06:13.299078 2078 log.cc:1196] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/fe4cc445dc944c52b2f70d287b79fa3b
I20260927 04:06:13.299396 2079 ts_tablet_manager.cc:1939] T fe4cc445dc944c52b2f70d287b79fa3b P 12ab387c486444ebb27c57f6f83e578b: Deleting consensus metadata
I20260927 04:06:13.299403 2078 ts_tablet_manager.cc:1939] T fe4cc445dc944c52b2f70d287b79fa3b P 0ed9147adf4c432db5f4fcf65aab4a20: Deleting consensus metadata
I20260927 04:06:13.299829 1577 catalog_manager.cc:4841] TS 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023): tablet fe4cc445dc944c52b2f70d287b79fa3b (table test-table [id=a66640af02b64315bf06f7f59d1bff11]) successfully deleted
I20260927 04:06:13.299870 1575 catalog_manager.cc:4841] TS 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793): tablet fe4cc445dc944c52b2f70d287b79fa3b (table test-table [id=a66640af02b64315bf06f7f59d1bff11]) successfully deleted
W20260927 04:06:13.301406 1577 connection.cc:535] client connection to 127.17.184.193:35897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 04:06:13.301532 1577 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 04:06:13.302078 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:13.301928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:35897 (local address 127.0.0.1:55332)
0927 04:06:13.301992 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:13.302034 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:35897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 04:06:13.302186 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 1785
W20260927 04:06:13.302196 1577 catalog_manager.cc:4584] TS 02d7a3620f864ca0bb7cd38b31c0f427 (127.17.184.193:35897): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe4cc445dc944c52b2f70d287b79fa3b: Network error: Client connection negotiation failed: client connection to 127.17.184.193:35897: connect: Connection refused (error 111)
I20260927 04:06:13.307071 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 1921
I20260927 04:06:13.312853 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 1558
I20260927 04:06:33.318604 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44953
--webserver_interface=127.17.184.254
--webserver_port=33353
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:33.415271 2081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:33.415534 2081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:33.415571 2081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:33.416966 2081 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 04:06:33.417038 2081 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 04:06:33.417059 2081 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 04:06:33.417078 2081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:33.417109 2081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:33.417140 2081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:33.417173 2081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:33.417204 2081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:33.419045 2081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=33353
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:33.419358 2081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:33.419606 2081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:33.426939 2087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:33.427175 2089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:33.427023 2086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:33.427201 2081 server_base.cc:1034] running on GCE node
I20260927 04:06:33.427820 2081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:33.428161 2081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:33.429345 2081 hybrid_clock.cc:648] HybridClock initialized: now 1790481993429315 us; error 43 us; skew 500 ppm
Sep 27 04:06:33 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481993, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:33.432282 2081 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:06:33.432885 2081 webserver.cc:460] Webserver started at http://127.17.184.254:33353/ using document root <none> and password file <none>
I20260927 04:06:33.433145 2081 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:33.433215 2081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:33.434701 2081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:33.435832 2096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:33.436079 2081 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:06:33.436218 2081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "a0233c61cf5e3c43f625c488551bcd86"
server_key_iv: "2f44e17c050daf890f7746783147ac49"
server_key_version: "encryptionkey@0"
I20260927 04:06:33.436616 2081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:33.459318 2081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:33.462405 2081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:33.462587 2081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:33.467804 2081 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44953
I20260927 04:06:33.467862 2148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44953 every 8 connection(s)
I20260927 04:06:33.468307 2081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:33.470682 2149 sys_catalog.cc:263] Verifying existing consensus state
I20260927 04:06:33.471441 2149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Bootstrap starting.
I20260927 04:06:33.475589 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 2081
I20260927 04:06:33.475914 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:35897
--local_ip_for_outbound_sockets=127.17.184.193
--tserver_master_addrs=127.17.184.254:44953
--webserver_port=40785
--webserver_interface=127.17.184.193
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:06:33.480953 2149 log.cc:824] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:33.483417 2149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:06:33.483729 2149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Bootstrap complete.
I20260927 04:06:33.486019 2149 raft_consensus.cc:348] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:06:33.486274 2149 raft_consensus.cc:729] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff842ae9f3b4bd2a33cf37bd61d6a8d, State: Initialized, Role: FOLLOWER
I20260927 04:06:33.486465 2149 consensus_queue.cc:260] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:06:33.486527 2149 raft_consensus.cc:388] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:33.486589 2149 raft_consensus.cc:482] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:33.486645 2149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 2 FOLLOWER]: Advancing to term 3
I20260927 04:06:33.488272 2149 raft_consensus.cc:504] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:06:33.488410 2149 leader_election.cc:302] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dff842ae9f3b4bd2a33cf37bd61d6a8d; no voters:
I20260927 04:06:33.488628 2149 leader_election.cc:288] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 04:06:33.488659 2153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 3 FOLLOWER]: Leader election won for term 3
I20260927 04:06:33.488772 2153 raft_consensus.cc:686] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [term 3 LEADER]: Becoming Leader. State: Replica: dff842ae9f3b4bd2a33cf37bd61d6a8d, State: Running, Role: LEADER
I20260927 04:06:33.488912 2149 sys_catalog.cc:564] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:33.488926 2153 consensus_queue.cc:237] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } }
I20260927 04:06:33.489326 2153 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dff842ae9f3b4bd2a33cf37bd61d6a8d. Latest consensus state: current_term: 3 leader_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } } }
I20260927 04:06:33.489323 2154 sys_catalog.cc:455] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dff842ae9f3b4bd2a33cf37bd61d6a8d" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44953 } } }
I20260927 04:06:33.489537 2153 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:33.489624 2154 sys_catalog.cc:458] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:33.489836 2162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:33.490823 2162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:06:33.491055 2162 catalog_manager.cc:1259] Loaded cluster ID: d76cfbdc2ba4425c80d8a804c91427ce
I20260927 04:06:33.491284 2162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:06:33.492724 2162 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:33.492975 2162 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Loaded TSK: 0
I20260927 04:06:33.493059 2162 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dff842ae9f3b4bd2a33cf37bd61d6a8d: Loaded TSK: 1
F20260927 04:06:33.493342 2162 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fea9aa7a9ad google::LogMessage::Fail() at ??:0
@ 0x7fea9aa7d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fea9aa7a4de google::LogMessage::Flush() at ??:0
@ 0x7fea9aa7ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fea9d1e7104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fea9d1e4d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7feaa29d81cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7feaa29d848e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7feaa29d8594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7feaa299ec48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7feaa29a8ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fea9c9480d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fea9c93d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fea9d7a16db start_thread at ??:0
@ 0x7fea9957861f clone at ??:0
W20260927 04:06:33.586313 2151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:33.586522 2151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:33.586549 2151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:33.588032 2151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:33.588109 2151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:33.588140 2151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:06:33.589802 2151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:35897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=40785
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:33.590091 2151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:33.590337 2151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:33.595763 2190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:33.595883 2191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:33.595763 2193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:33.595971 2151 server_base.cc:1034] running on GCE node
I20260927 04:06:33.596197 2151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:33.596437 2151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:33.597621 2151 hybrid_clock.cc:648] HybridClock initialized: now 1790481993597592 us; error 47 us; skew 500 ppm
Sep 27 04:06:33 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481993, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:33.600270 2151 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:06:33.600737 2151 webserver.cc:460] Webserver started at http://127.17.184.193:40785/ using document root <none> and password file <none>
I20260927 04:06:33.600982 2151 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:33.601047 2151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:33.602500 2151 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:33.603433 2200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:33.603650 2151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:06:33.603729 2151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "02d7a3620f864ca0bb7cd38b31c0f427"
format_stamp: "Formatted at 2026-09-27 04:05:51 on dist-test-slave-zqrn"
server_key: "9f9f72dd24e49bc50d154f85377ff794"
server_key_iv: "c8d44fe05389f355ca004c5feead5f72"
server_key_version: "encryptionkey@0"
I20260927 04:06:33.604058 2151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:33.622253 2151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:33.625464 2151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:33.625658 2151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:33.625941 2151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:33.626480 2207 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 04:06:33.628886 2151 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 04:06:33.628979 2151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260927 04:06:33.629041 2151 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 04:06:33.629769 2151 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 04:06:33.629828 2151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260927 04:06:33.629910 2207 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Bootstrap starting.
I20260927 04:06:33.637969 2151 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:35897
I20260927 04:06:33.638417 2151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:06:33.638962 2314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:35897 every 8 connection(s)
I20260927 04:06:33.642020 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 2151
I20260927 04:06:33.642062 2207 log.cc:824] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:33.642503 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:33023
--local_ip_for_outbound_sockets=127.17.184.194
--tserver_master_addrs=127.17.184.254:44953
--webserver_port=39155
--webserver_interface=127.17.184.194
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:06:33.645318 2207 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 04:06:33.645699 2207 tablet_bootstrap.cc:492] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Bootstrap complete.
I20260927 04:06:33.647109 2207 ts_tablet_manager.cc:1397] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.012s
I20260927 04:06:33.648839 2207 raft_consensus.cc:348] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:33.649209 2207 raft_consensus.cc:729] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d7a3620f864ca0bb7cd38b31c0f427, State: Initialized, Role: FOLLOWER
I20260927 04:06:33.649402 2207 consensus_queue.cc:260] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:33.649720 2207 ts_tablet_manager.cc:1428] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260927 04:06:33.706481 2317 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:06:33.639287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:52541)
0927 04:06:33.639542 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.639591 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:33.640085 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:33.706258 (+ 66173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":107,"threads_started":1}
W20260927 04:06:33.706790 2315 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:44953 (0 consecutive failures): Network error: Failed to ping master at 127.17.184.254:44953: Client connection negotiation failed: client connection to 127.17.184.254:44953: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 04:06:33.707191 2317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:33.707052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:41567)
0927 04:06:33.707113 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.707162 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 04:06:33.707582 2317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:33.707475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:56139)
0927 04:06:33.707522 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.707555 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:06:33.707728 2315 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 04:06:33.732321 2319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:33.732548 2319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:33.732584 2319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:33.733868 2319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:33.733917 2319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:33.733947 2319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:06:33.735628 2319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:33023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=39155
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:33.735895 2319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:33.736140 2319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:33.741670 2325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:33.741734 2319 server_base.cc:1034] running on GCE node
W20260927 04:06:33.741778 2328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:33.741626 2326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:33.742059 2319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:33.742292 2319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:33.743458 2319 hybrid_clock.cc:648] HybridClock initialized: now 1790481993743443 us; error 32 us; skew 500 ppm
Sep 27 04:06:33 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481993, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:33.746158 2319 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:06:33.746649 2319 webserver.cc:460] Webserver started at http://127.17.184.194:39155/ using document root <none> and password file <none>
I20260927 04:06:33.746951 2319 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:33.747022 2319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:33.748557 2319 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:33.749346 2335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:33.749571 2319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:33.749692 2319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "0ed9147adf4c432db5f4fcf65aab4a20"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "73092d704412955dd5469ebc08935a74"
server_key_iv: "c3efeb6ea77d5391485e4748c8427e76"
server_key_version: "encryptionkey@0"
I20260927 04:06:33.750062 2319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:33.770542 2319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:33.773001 2319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:33.773200 2319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:33.773468 2319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:33.773874 2319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:33.774068 2319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:33.774148 2319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:33.774178 2319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:33.780336 2319 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33023
I20260927 04:06:33.780383 2448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33023 every 8 connection(s)
I20260927 04:06:33.780767 2319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:06:33.782150 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:33.781760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:36931)
0927 04:06:33.781919 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.781991 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
W20260927 04:06:33.782577 2449 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:44953 (0 consecutive failures): Network error: Failed to ping master at 127.17.184.254:44953: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
I20260927 04:06:33.782912 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:33.782810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:45265)
0927 04:06:33.782853 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.782880 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 04:06:33.783280 2451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:33.783125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:45111)
0927 04:06:33.783223 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.783258 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 04:06:33.783463 2449 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 04:06:33.789042 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 2319
I20260927 04:06:33.789356 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:34793
--local_ip_for_outbound_sockets=127.17.184.195
--tserver_master_addrs=127.17.184.254:44953
--webserver_port=35745
--webserver_interface=127.17.184.195
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:33.872594 2452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:33.872814 2452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:33.872838 2452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:33.874070 2452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:33.874111 2452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:33.874140 2452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:06:33.875661 2452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:34793
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=35745
--tserver_master_addrs=127.17.184.254:44953
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:33.875919 2452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:33.876152 2452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:33.881487 2457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:33.881489 2458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:33.881853 2460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:33.882560 2452 server_base.cc:1034] running on GCE node
I20260927 04:06:33.882855 2452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:33.883147 2452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:33.884285 2452 hybrid_clock.cc:648] HybridClock initialized: now 1790481993884262 us; error 40 us; skew 500 ppm
Sep 27 04:06:33 dist-test-slave-zqrn krb5kdc[1010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481993, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:33.887383 2452 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:06:33.887934 2452 webserver.cc:460] Webserver started at http://127.17.184.195:35745/ using document root <none> and password file <none>
I20260927 04:06:33.888226 2452 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:33.888300 2452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:33.890271 2452 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:06:33.891251 2467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:33.891491 2452 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:33.891672 2452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "12ab387c486444ebb27c57f6f83e578b"
format_stamp: "Formatted at 2026-09-27 04:05:52 on dist-test-slave-zqrn"
server_key: "5e6038173d98851baea9e666b8bf895b"
server_key_iv: "c09bfa6eacee44daf37df46878976c19"
server_key_version: "encryptionkey@0"
I20260927 04:06:33.892068 2452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:33.905908 2452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:33.908288 2452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:33.908437 2452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:33.908694 2452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:33.909003 2452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:33.909055 2452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:33.909107 2452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:33.909135 2452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:33.915501 2452 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:34793
I20260927 04:06:33.915671 2580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:34793 every 8 connection(s)
I20260927 04:06:33.915962 2452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:06:33.917092 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:33.916657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:52537)
0927 04:06:33.916821 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.916937 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
W20260927 04:06:33.917550 2581 heartbeater.cc:644] Failed to heartbeat to 127.17.184.254:44953 (0 consecutive failures): Network error: Failed to ping master at 127.17.184.254:44953: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
I20260927 04:06:33.917908 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:33.917814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:44863)
0927 04:06:33.917858 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.917888 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 04:06:33.918192 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:33.918128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:43585)
0927 04:06:33.918153 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:33.918173 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260927 04:06:33.918318 2581 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 04:06:33.924939 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 2452
I20260927 04:06:34.708323 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:34.708093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:47681)
0927 04:06:34.708215 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:34.708258 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260927 04:06:34.784060 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:34.783815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:46215)
0927 04:06:34.783961 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:34.784002 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260927 04:06:34.865888 2587 raft_consensus.cc:482] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:34.866027 2587 raft_consensus.cc:504] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:34.866338 2587 leader_election.cc:288] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
Sep 27 04:06:34 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481993, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.195@KRBTEST.COM
Sep 27 04:06:34 dist-test-slave-zqrn krb5kdc[1010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790481993, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.194@KRBTEST.COM
I20260927 04:06:34.877985 2585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:34.866536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33023 (local address 127.17.184.193:46769)
0927 04:06:34.866580 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:34.866593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:34.866649 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:34.867669 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:34.867670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:34.868050 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:34.868257 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:34.868264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:34.869890 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:34.869891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:34.870733 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:34.870735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:34.871480 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:34.871482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:34.871549 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:34.872261 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:34.872266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:34.873972 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:34.876374 (+ 2402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:34.876375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:34.876375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:34.876568 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:34.876818 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:34.876819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:34.876819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:34.876886 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:34.877333 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:34.877341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:34.877500 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 04:06:34.877614 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 04:06:34.877822 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":33}
I20260927 04:06:34.878003 2590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:34.866668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:37375 (local address 127.17.184.195:34793)
0927 04:06:34.866818 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:34.866830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:34.866854 (+ 24us) server_negotiation.cc:407] Connection header received
0927 04:06:34.867350 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:34.867352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:34.867408 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:34.867565 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:34.868387 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:34.869786 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:34.870821 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:34.871290 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:34.871411 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:34.873760 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:34.873764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:34.873765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:34.873779 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:34.876011 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:34.876501 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:34.876504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:34.876504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:34.876621 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:34.876892 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:34.876893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:34.876893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:34.877249 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:34.877333 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:34.877732 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 04:06:34.877889 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260927 04:06:34.878014 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:34.866659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:46769 (local address 127.17.184.194:33023)
0927 04:06:34.866819 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:34.866828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:34.866857 (+ 29us) server_negotiation.cc:407] Connection header received
0927 04:06:34.867383 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:34.867386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:34.867425 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:34.867608 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:34.868431 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:34.869787 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:34.870861 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:34.871373 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:34.871528 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:34.874073 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:34.874077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:34.874078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:34.874102 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:34.876296 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:34.876643 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:34.876645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:34.876646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:34.876746 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:34.876948 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:34.876949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:34.876950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:34.877249 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:34.877333 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:34.877740 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 04:06:34.877888 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":55,"threads_started":1}
I20260927 04:06:34.877985 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:34.866536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:34793 (local address 127.17.184.193:37375)
0927 04:06:34.866745 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:34.866756 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:34.866794 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:34.867615 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:34.867619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:34.868049 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:34.868257 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:34.868264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:34.869922 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:34.869923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:34.870731 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:34.870733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:34.871380 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:34.871381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:34.871493 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:34.872063 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:34.872071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:34.873629 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:34.876126 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:34.876133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:34.876134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:34.876410 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:34.876696 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:34.876696 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:34.876697 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:34.876828 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:34.877333 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:34.877341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:34.877500 (+ 159us) client_negotiation.cc:769] Sending connection context
0927 04:06:34.877670 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 04:06:34.877881 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":238,"thread_start_us":129,"threads_started":1}
W20260927 04:06:34.879245 2203 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
W20260927 04:06:34.879443 2201 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
I20260927 04:06:34.879498 2201 leader_election.cc:302] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427; no voters: 0ed9147adf4c432db5f4fcf65aab4a20, 12ab387c486444ebb27c57f6f83e578b
I20260927 04:06:34.879619 2587 raft_consensus.cc:2726] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:06:34.918844 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:34.918645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:58603)
0927 04:06:34.918763 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:34.918808 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260927 04:06:35.709112 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:35.708887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:40163)
0927 04:06:35.709022 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:35.709061 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260927 04:06:35.784777 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:35.784533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:53069)
0927 04:06:35.784689 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:35.784728 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260927 04:06:35.919631 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:35.919402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:41417)
0927 04:06:35.919552 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:35.919586 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260927 04:06:36.675859 2595 raft_consensus.cc:482] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:36.675961 2595 raft_consensus.cc:504] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:36.676343 2595 leader_election.cc:288] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
W20260927 04:06:36.676769 2203 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
W20260927 04:06:36.676862 2201 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
I20260927 04:06:36.676934 2201 leader_election.cc:302] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427; no voters: 0ed9147adf4c432db5f4fcf65aab4a20, 12ab387c486444ebb27c57f6f83e578b
I20260927 04:06:36.677021 2595 raft_consensus.cc:2726] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:06:36.710007 2596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:36.709753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:33355)
0927 04:06:36.709913 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:36.709957 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260927 04:06:36.785811 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:36.785572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:53025)
0927 04:06:36.785722 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:36.785767 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260927 04:06:36.920387 2598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:36.920158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:52193)
0927 04:06:36.920299 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:36.920345 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260927 04:06:37.710981 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:37.710545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:49569)
0927 04:06:37.710832 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:37.710883 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":188,"threads_started":1}
I20260927 04:06:37.786520 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:37.786303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:38909)
0927 04:06:37.786432 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:37.786463 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260927 04:06:37.921258 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:37.921039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:58475)
0927 04:06:37.921167 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:37.921213 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260927 04:06:38.711683 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:38.711466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:42229)
0927 04:06:38.711596 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:38.711633 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260927 04:06:38.787472 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:38.787218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:35711)
0927 04:06:38.787368 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:38.787415 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260927 04:06:38.922165 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:38.921952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:45305)
0927 04:06:38.922079 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:38.922119 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260927 04:06:39.712678 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:39.712273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:47245)
0927 04:06:39.712585 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:39.712629 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":225,"threads_started":1}
I20260927 04:06:39.788406 2606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:39.788140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:52215)
0927 04:06:39.788313 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:39.788359 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260927 04:06:39.923034 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:39.922739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:49851)
0927 04:06:39.922890 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:39.922987 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":73,"threads_started":1}
I20260927 04:06:40.040375 2608 raft_consensus.cc:482] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:40.040478 2608 raft_consensus.cc:504] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:40.040704 2608 leader_election.cc:288] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
W20260927 04:06:40.041225 2201 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
W20260927 04:06:40.041301 2203 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
I20260927 04:06:40.041400 2203 leader_election.cc:302] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427; no voters: 0ed9147adf4c432db5f4fcf65aab4a20, 12ab387c486444ebb27c57f6f83e578b
I20260927 04:06:40.041486 2608 raft_consensus.cc:2726] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:06:40.713568 2609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:40.713317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:44929)
0927 04:06:40.713464 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:40.713519 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260927 04:06:40.789281 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:40.788987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:45321)
0927 04:06:40.789163 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:40.789205 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260927 04:06:40.923803 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:40.923546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:47789)
0927 04:06:40.923709 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:40.923756 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260927 04:06:41.714340 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:41.714102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:58591)
0927 04:06:41.714248 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:41.714293 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":63,"threads_started":1}
I20260927 04:06:41.790093 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:41.789864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:38713)
0927 04:06:41.790002 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:41.790045 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260927 04:06:41.924530 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:41.924291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:43753)
0927 04:06:41.924443 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:41.924483 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260927 04:06:42.715216 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:42.714996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:37307)
0927 04:06:42.715121 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:42.715166 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260927 04:06:42.790956 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:42.790682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:42749)
0927 04:06:42.790852 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:42.790910 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":62,"threads_started":1}
I20260927 04:06:42.925308 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:42.925037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:58447)
0927 04:06:42.925217 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:42.925262 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260927 04:06:43.715911 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:43.715674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:42135)
0927 04:06:43.715823 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:43.715862 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260927 04:06:43.791806 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:43.791559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:33941)
0927 04:06:43.791722 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:43.791760 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":83,"threads_started":1}
I20260927 04:06:43.926026 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:43.925769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:58229)
0927 04:06:43.925903 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:43.925946 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260927 04:06:44.333750 2621 raft_consensus.cc:482] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:44.333848 2621 raft_consensus.cc:504] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:44.334049 2621 leader_election.cc:288] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
W20260927 04:06:44.334604 2203 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
W20260927 04:06:44.334692 2201 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
I20260927 04:06:44.334728 2201 leader_election.cc:302] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427; no voters: 0ed9147adf4c432db5f4fcf65aab4a20, 12ab387c486444ebb27c57f6f83e578b
I20260927 04:06:44.334857 2621 raft_consensus.cc:2726] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:06:44.716611 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:44.716396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:48601)
0927 04:06:44.716527 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:44.716559 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260927 04:06:44.792577 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:44.792298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:55639)
0927 04:06:44.792430 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:44.792513 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260927 04:06:44.926731 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:44.926488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:51099)
0927 04:06:44.926646 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:44.926686 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":66,"threads_started":1}
I20260927 04:06:45.717414 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:45.717132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:51997)
0927 04:06:45.717318 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:45.717365 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
I20260927 04:06:45.793354 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:45.793129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:57671)
0927 04:06:45.793262 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:45.793306 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260927 04:06:45.927537 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:45.927317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:33711)
0927 04:06:45.927447 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:45.927491 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260927 04:06:46.718096 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:46.717892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:52077)
0927 04:06:46.718013 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:46.718046 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260927 04:06:46.794214 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:46.793933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:48439)
0927 04:06:46.794118 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:46.794163 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
I20260927 04:06:46.928279 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:46.928019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:48575)
0927 04:06:46.928175 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:46.928227 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":65,"threads_started":1}
I20260927 04:06:47.204617 2631 raft_consensus.cc:482] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:47.204715 2631 raft_consensus.cc:504] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ed9147adf4c432db5f4fcf65aab4a20" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33023 } } peers { permanent_uuid: "12ab387c486444ebb27c57f6f83e578b" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 34793 } } peers { permanent_uuid: "02d7a3620f864ca0bb7cd38b31c0f427" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 35897 } }
I20260927 04:06:47.204857 2631 leader_election.cc:288] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023), 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793)
W20260927 04:06:47.205245 2203 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ed9147adf4c432db5f4fcf65aab4a20 (127.17.184.194:33023): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
W20260927 04:06:47.205328 2201 leader_election.cc:341] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 12ab387c486444ebb27c57f6f83e578b (127.17.184.195:34793): Not found: Tablet not found: fe4cc445dc944c52b2f70d287b79fa3b
I20260927 04:06:47.205395 2201 leader_election.cc:302] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02d7a3620f864ca0bb7cd38b31c0f427; no voters: 0ed9147adf4c432db5f4fcf65aab4a20, 12ab387c486444ebb27c57f6f83e578b
I20260927 04:06:47.205456 2631 raft_consensus.cc:2726] T fe4cc445dc944c52b2f70d287b79fa3b P 02d7a3620f864ca0bb7cd38b31c0f427 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 04:06:47.718827 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:47.718611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:47543)
0927 04:06:47.718738 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:47.718777 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260927 04:06:47.795091 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:47.794787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:54099)
0927 04:06:47.794977 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:47.795018 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":62,"threads_started":1}
I20260927 04:06:47.928985 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:47.928780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:40387)
0927 04:06:47.928905 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:47.928942 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260927 04:06:48.719691 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:48.719445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.193:32793)
0927 04:06:48.719595 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:48.719644 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260927 04:06:48.795933 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:48.795688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.194:45847)
0927 04:06:48.795838 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:48.795884 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":91,"threads_started":1}
I20260927 04:06:48.929795 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:48.929548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44953 (local address 127.17.184.195:46231)
0927 04:06:48.929705 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:48.929749 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.254:44953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260927 04:06:48.933674 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 2151
I20260927 04:06:48.938289 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 2319
I20260927 04:06:48.943601 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 2452
2026-09-27T04:06:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58444 ms)
[----------] 30 tests from SecurityITest (235915 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:06:49 dist-test-slave-zqrn krb5kdc[2640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:06:49 dist-test-slave-zqrn krb5kdc[2640](info): set up 2 sockets
Sep 27 04:06:49 dist-test-slave-zqrn krb5kdc[2640](info): commencing operation
krb5kdc: starting...
W20260927 04:06:50.031168 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:06:50.159539 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44095
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:50.240127 2653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:50.240392 2653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:50.240429 2653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:50.241585 2653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:50.241662 2653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:50.241685 2653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:50.241707 2653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:50.241735 2653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:50.243252 2653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:50.243542 2653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:50.243803 2653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:50.249187 2658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:50.249280 2653 server_base.cc:1034] running on GCE node
W20260927 04:06:50.249212 2661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:50.249243 2659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:50.250000 2653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:50.250419 2653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:50.251672 2653 hybrid_clock.cc:648] HybridClock initialized: now 1790482010251652 us; error 37 us; skew 500 ppm
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:50.254289 2653 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:06:50.254822 2653 webserver.cc:460] Webserver started at http://127.17.184.254:33127/ using document root <none> and password file <none>
I20260927 04:06:50.255103 2653 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:50.255177 2653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:50.255324 2653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:50.256397 2653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "c42241e1934a4aef948eba3255234319"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "cdd49e1c44c08c042cadcb6614d5a1f8"
server_key_iv: "506f7f2083a44818f6bff4fac7319c19"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.256760 2653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "c42241e1934a4aef948eba3255234319"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "cdd49e1c44c08c042cadcb6614d5a1f8"
server_key_iv: "506f7f2083a44818f6bff4fac7319c19"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.258116 2653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:50.259143 2668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.259341 2653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:50.259451 2653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "c42241e1934a4aef948eba3255234319"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "cdd49e1c44c08c042cadcb6614d5a1f8"
server_key_iv: "506f7f2083a44818f6bff4fac7319c19"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.259580 2653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:50.273375 2653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:50.275847 2653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:50.276023 2653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:50.280040 2653 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44095
I20260927 04:06:50.280084 2720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44095 every 8 connection(s)
I20260927 04:06:50.280457 2653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:50.281455 2721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:50.283892 2721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319: Bootstrap starting.
I20260927 04:06:50.284466 2721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:50.284752 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 2653
I20260927 04:06:50.284766 2721 log.cc:824] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:50.284930 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:06:50.285164 18147 external_mini_cluster.cc:1428] Setting key e7feb4366eeaa62e0687e14c3eff8bd2
I20260927 04:06:50.285507 2721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319: No bootstrap required, opened a new log
I20260927 04:06:50.287235 2721 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42241e1934a4aef948eba3255234319" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44095 } }
I20260927 04:06:50.287367 2721 raft_consensus.cc:374] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:50.287389 2721 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42241e1934a4aef948eba3255234319, State: Initialized, Role: FOLLOWER
I20260927 04:06:50.287515 2721 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42241e1934a4aef948eba3255234319" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44095 } }
I20260927 04:06:50.287612 2721 raft_consensus.cc:388] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:50.287679 2721 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:50.287734 2721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:50.288610 2721 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42241e1934a4aef948eba3255234319" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44095 } }
I20260927 04:06:50.288731 2721 leader_election.cc:302] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c42241e1934a4aef948eba3255234319; no voters:
I20260927 04:06:50.288921 2721 leader_election.cc:288] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:06:50.288955 2726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:50.289059 2726 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [term 1 LEADER]: Becoming Leader. State: Replica: c42241e1934a4aef948eba3255234319, State: Running, Role: LEADER
I20260927 04:06:50.289173 2721 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:50.289206 2726 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42241e1934a4aef948eba3255234319" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44095 } }
I20260927 04:06:50.289522 2726 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42241e1934a4aef948eba3255234319. Latest consensus state: current_term: 1 leader_uuid: "c42241e1934a4aef948eba3255234319" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42241e1934a4aef948eba3255234319" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44095 } } }
I20260927 04:06:50.289538 2728 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c42241e1934a4aef948eba3255234319" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42241e1934a4aef948eba3255234319" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44095 } } }
I20260927 04:06:50.289690 2726 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:50.289979 2728 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319 [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:50.290517 2735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:50.291041 2735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:06:50.293615 2735 catalog_manager.cc:1347] Generated new cluster ID: eebffdcaff0549a7b4965f5cffbe2023
I20260927 04:06:50.293668 2735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:50.299274 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.285674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.17.184.254:44095)
0927 04:06:50.285853 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.285864 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.285899 (+ 35us) server_negotiation.cc:407] Connection header received
0927 04:06:50.287258 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.287267 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.287327 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.287577 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:50.288376 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.289873 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.291129 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.291711 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.291835 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.295025 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:50.295030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:50.295030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:50.295044 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:50.297123 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.297584 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.297588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.297588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.297731 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.298011 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.298012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.298013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.298304 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:50.298604 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.298690 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.299053 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260927 04:06:50.322618 2735 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:06:50.323279 2735 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:50.325811 2735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319: Generated new TSK 0
I20260927 04:06:50.325999 2735 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:06:50.358379 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44095
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:50.454811 2747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:50.455127 2747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:50.455175 2747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:50.456390 2747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:50.456454 2747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:50.456499 2747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:06:50.458026 2747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:50.458309 2747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:50.458544 2747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:50.463729 2753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:50.463729 2755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:50.463755 2752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:50.464059 2747 server_base.cc:1034] running on GCE node
I20260927 04:06:50.464277 2747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:50.464520 2747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:50.465682 2747 hybrid_clock.cc:648] HybridClock initialized: now 1790482010465644 us; error 51 us; skew 500 ppm
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:50.468257 2747 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:06:50.468694 2747 webserver.cc:460] Webserver started at http://127.17.184.193:43835/ using document root <none> and password file <none>
I20260927 04:06:50.468920 2747 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:50.468988 2747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:50.469141 2747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:50.470212 2747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "6db638c41ce84f9ab89688ba8a85762b"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "43262cf47f1bb3704103e5df5633e433"
server_key_iv: "5e263f69c11e870cfadf9f0f5dda3b6e"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.470572 2747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "6db638c41ce84f9ab89688ba8a85762b"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "43262cf47f1bb3704103e5df5633e433"
server_key_iv: "5e263f69c11e870cfadf9f0f5dda3b6e"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.472008 2747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:50.472826 2762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.473048 2747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:50.473151 2747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "6db638c41ce84f9ab89688ba8a85762b"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "43262cf47f1bb3704103e5df5633e433"
server_key_iv: "5e263f69c11e870cfadf9f0f5dda3b6e"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.473239 2747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:50.509927 2747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:50.512063 2747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:50.512243 2747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:50.512492 2747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:50.512800 2747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:50.512853 2747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.512897 2747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:50.512924 2747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.518865 2747 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:43285
I20260927 04:06:50.518929 2875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:43285 every 8 connection(s)
I20260927 04:06:50.519256 2747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:06:50.524439 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 2747
I20260927 04:06:50.524528 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:06:50.524763 18147 external_mini_cluster.cc:1428] Setting key 690c06de5531995a6b29cff57c19ce19
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:50.529208 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.519956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44095 (local address 127.17.184.193:48267)
0927 04:06:50.520159 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.520202 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.520677 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.520882 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.520885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.521445 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:50.521647 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.521652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.522312 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.522313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.523262 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.523265 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.523614 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.523615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.523720 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.524179 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:50.524187 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:50.525892 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:50.527718 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:50.527726 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:50.527728 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:50.528078 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:50.528316 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:50.528317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:50.528317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:50.528417 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:50.528705 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:50.528709 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:50.528825 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.528911 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.529047 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":128,"threads_started":1}
I20260927 04:06:50.529201 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.519995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:48267 (local address 127.17.184.254:44095)
0927 04:06:50.520069 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.520070 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.520291 (+ 221us) server_negotiation.cc:407] Connection header received
0927 04:06:50.520760 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.520761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.520770 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.520829 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:50.521726 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.522225 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.523340 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.523542 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.523597 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.525984 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:50.525986 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:50.525986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:50.525995 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:50.527630 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.528188 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.528189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.528189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.528249 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.528482 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.528484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.528484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.528635 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:50.528707 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.528956 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.529052 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:06:50.529778 2876 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44095
I20260927 04:06:50.529886 2876 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:50.530071 2876 heartbeater.cc:511] Master 127.17.184.254:44095 requested a full tablet report, sending...
I20260927 04:06:50.530607 2685 ts_manager.cc:194] Registered new tserver with Master: 6db638c41ce84f9ab89688ba8a85762b (127.17.184.193:43285)
I20260927 04:06:50.531306 2685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:48267
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:06:50.551721 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44095
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:50.631489 2881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:50.631696 2881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:50.631719 2881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:50.632894 2881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:50.632934 2881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:50.632995 2881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:06:50.634524 2881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:50.634780 2881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:50.635025 2881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:50.639917 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:50.639935 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:50.639909 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:50.640208 2881 server_base.cc:1034] running on GCE node
I20260927 04:06:50.640380 2881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:50.640570 2881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:50.641706 2881 hybrid_clock.cc:648] HybridClock initialized: now 1790482010641699 us; error 30 us; skew 500 ppm
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:50.644091 2881 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:06:50.644451 2881 webserver.cc:460] Webserver started at http://127.17.184.194:36855/ using document root <none> and password file <none>
I20260927 04:06:50.644652 2881 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:50.644692 2881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:50.644778 2881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:50.645784 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "e42f87998b6f46c2b807f640e8df2a1d"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "9fb949967a549f3186eea1c1cd7df994"
server_key_iv: "f6d179524fff90fb9586d53a017a2781"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.646136 2881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "e42f87998b6f46c2b807f640e8df2a1d"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "9fb949967a549f3186eea1c1cd7df994"
server_key_iv: "f6d179524fff90fb9586d53a017a2781"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.647431 2881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:50.648095 2896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.648263 2881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:06:50.648367 2881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "e42f87998b6f46c2b807f640e8df2a1d"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "9fb949967a549f3186eea1c1cd7df994"
server_key_iv: "f6d179524fff90fb9586d53a017a2781"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.648440 2881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:50.669932 2881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:50.671913 2881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:50.672096 2881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:50.672348 2881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:50.672649 2881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:50.672704 2881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.672744 2881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:50.672775 2881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.678545 2881 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:34275
I20260927 04:06:50.678632 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:34275 every 8 connection(s)
I20260927 04:06:50.678995 2881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:50.686606 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 2881
I20260927 04:06:50.686721 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:06:50.686990 18147 external_mini_cluster.cc:1428] Setting key b59363bc507eb51bacc48bebe757d3be
I20260927 04:06:50.688364 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.679677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:39521 (local address 127.17.184.254:44095)
0927 04:06:50.679725 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.679727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.679937 (+ 210us) server_negotiation.cc:407] Connection header received
0927 04:06:50.680422 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.680423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.680432 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.680489 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:50.681372 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.681892 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.682849 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.683138 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.683202 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.685255 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:50.685256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:50.685257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:50.685264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:50.686767 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.687264 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.687266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.687266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.687326 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.687664 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.687666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.687666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.687799 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:50.687886 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.688190 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.688271 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:06:50.688580 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.679636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44095 (local address 127.17.184.194:39521)
0927 04:06:50.679799 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.679840 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.680334 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.680540 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.680543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.681097 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:50.681285 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.681289 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.681972 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.681973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.682748 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.682750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.683218 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.683219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.683318 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.683740 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:50.683747 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:50.685143 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:50.686845 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:50.686851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:50.686853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:50.687170 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:50.687401 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:50.687402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:50.687403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:50.687538 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:50.687878 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:50.687885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:50.688039 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.688169 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.688331 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260927 04:06:50.689097 3010 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44095
I20260927 04:06:50.689203 3010 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:50.689390 3010 heartbeater.cc:511] Master 127.17.184.254:44095 requested a full tablet report, sending...
I20260927 04:06:50.689834 2685 ts_manager.cc:194] Registered new tserver with Master: e42f87998b6f46c2b807f640e8df2a1d (127.17.184.194:34275)
I20260927 04:06:50.690263 2685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:39521
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:06:50.714165 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44095
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:50.796330 3015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:50.796597 3015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:50.796636 3015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:50.797933 3015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:50.797974 3015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:50.798002 3015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:06:50.799528 3015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:50.799784 3015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:50.800024 3015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:50.805217 3020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:50.805261 3015 server_base.cc:1034] running on GCE node
W20260927 04:06:50.805209 3021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:50.805217 3023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:50.805706 3015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:50.805950 3015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:50.807113 3015 hybrid_clock.cc:648] HybridClock initialized: now 1790482010807102 us; error 24 us; skew 500 ppm
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:50.809505 3015 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:06:50.809984 3015 webserver.cc:460] Webserver started at http://127.17.184.195:46785/ using document root <none> and password file <none>
I20260927 04:06:50.810220 3015 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:50.810287 3015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:50.810441 3015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:50.811515 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "dbe8694d6bfe49c7afe7e70f0d20a414"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "ab4603bbfa3e797490f8820fe2962aa2"
server_key_iv: "916880dc783888e617f7c1948d4d0109"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.811868 3015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "dbe8694d6bfe49c7afe7e70f0d20a414"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "ab4603bbfa3e797490f8820fe2962aa2"
server_key_iv: "916880dc783888e617f7c1948d4d0109"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.813194 3015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:50.813937 3030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.814126 3015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:50.814229 3015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "dbe8694d6bfe49c7afe7e70f0d20a414"
format_stamp: "Formatted at 2026-09-27 04:06:50 on dist-test-slave-zqrn"
server_key: "ab4603bbfa3e797490f8820fe2962aa2"
server_key_iv: "916880dc783888e617f7c1948d4d0109"
server_key_version: "encryptionkey@0"
I20260927 04:06:50.814303 3015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:50.834821 3015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:50.838712 3015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:50.838876 3015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:50.839171 3015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:50.839496 3015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:50.839551 3015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.839593 3015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:50.839620 3015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:50.845712 3015 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:43399
I20260927 04:06:50.845774 3143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:43399 every 8 connection(s)
I20260927 04:06:50.846132 3015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:06:50.849282 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 3015
I20260927 04:06:50.849395 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:06:50.849627 18147 external_mini_cluster.cc:1428] Setting key 816c2991d014535ebad2a825c8bc0088
Sep 27 04:06:50 dist-test-slave-zqrn krb5kdc[2640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482010, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:50.855991 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.846929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:57733 (local address 127.17.184.254:44095)
0927 04:06:50.846995 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.846998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.847222 (+ 224us) server_negotiation.cc:407] Connection header received
0927 04:06:50.847672 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.847673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.847685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.847753 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:50.848661 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.849255 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.850573 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.850791 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.850881 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.853158 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:50.853175 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:50.853175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:50.853183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:50.854531 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.854987 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.854988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.854988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.855054 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.855312 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.855313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.855313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.855435 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:50.855508 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.855817 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.855898 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 04:06:50.856207 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.846893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44095 (local address 127.17.184.195:57733)
0927 04:06:50.847093 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.847129 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.847586 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.847805 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.847809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.848376 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:50.848566 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.848570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.849377 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.849378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.850489 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.850492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.850924 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.850926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.851058 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.851637 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:50.851651 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:50.853062 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:50.854632 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:50.854639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:50.854641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:50.854867 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:50.855123 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:50.855124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:50.855125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:50.855238 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:50.855508 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:50.855515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:50.855679 (+ 164us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.855781 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.855941 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260927 04:06:50.856799 3144 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44095
I20260927 04:06:50.856899 3144 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:50.857048 3144 heartbeater.cc:511] Master 127.17.184.254:44095 requested a full tablet report, sending...
I20260927 04:06:50.857424 2685 ts_manager.cc:194] Registered new tserver with Master: dbe8694d6bfe49c7afe7e70f0d20a414 (127.17.184.195:43399)
I20260927 04:06:50.857812 2685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:57733
I20260927 04:06:50.861545 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:06:50.868635 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.862989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.17.184.254:44095)
0927 04:06:50.863045 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.863047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.863170 (+ 123us) server_negotiation.cc:407] Connection header received
0927 04:06:50.863218 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.863218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.863226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.863276 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:50.863818 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.864287 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.865017 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.865196 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.865244 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.866529 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:50.866530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:50.866530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:50.866535 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:50.867721 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.868014 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.868014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.868015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.868061 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:50.868235 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:50.868236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:50.868236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:50.868362 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:50.868434 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.868522 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.868573 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:06:50.870662 2685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:06:50.872386 2685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:06:50.879259 3157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.874746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:43285 (local address 127.0.0.1:50192)
0927 04:06:50.875126 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.875137 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.875195 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.875571 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.875572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.875575 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:50.875611 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.875612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.876928 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.876930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.877999 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.878002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.878886 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.878887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.879029 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.879045 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.879120 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.879170 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":73,"thread_start_us":64,"threads_started":1}
I20260927 04:06:50.879410 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.874744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:34275 (local address 127.0.0.1:41952)
0927 04:06:50.874968 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.875012 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.875104 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.875384 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.875388 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.875391 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:50.875466 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.875468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.876874 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.876875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.878236 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.878238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.879150 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.879151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.879228 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.879232 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.879289 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.879335 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":55,"spinlock_wait_cycles":768,"thread_start_us":103,"threads_started":1}
I20260927 04:06:50.879418 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.874944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.17.184.194:34275)
0927 04:06:50.875204 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.875209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.875220 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:06:50.875269 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.875271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.875289 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.875404 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:50.875538 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.876780 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.878318 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.879049 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.879148 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.879198 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.879305 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.879341 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260927 04:06:50.879571 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.874957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.17.184.193:43285)
0927 04:06:50.875234 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.875238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.875245 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:06:50.875276 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.875277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.875289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.875681 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:50.875766 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.876815 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.878090 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.878796 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.878880 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.878965 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.879378 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.879422 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":81,"threads_started":1}
I20260927 04:06:50.880108 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.874753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:43399 (local address 127.0.0.1:36340)
0927 04:06:50.875273 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.875282 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.875319 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.875769 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.875770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.875772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:50.875823 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.875825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.876827 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.876828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.878222 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.878225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.879876 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.879877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.879940 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.879944 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.880002 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.880045 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":58,"thread_start_us":99,"threads_started":1}
I20260927 04:06:50.880481 2810 tablet_service.cc:1467] Processing CreateTablet for tablet a1f964e924a84f749ba662d1d8abe98f (DEFAULT_TABLE table=test-table [id=5c1834ca6a5b4cfcb0ec6c3e2762879e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:50.880496 2944 tablet_service.cc:1467] Processing CreateTablet for tablet a1f964e924a84f749ba662d1d8abe98f (DEFAULT_TABLE table=test-table [id=5c1834ca6a5b4cfcb0ec6c3e2762879e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:50.880822 2810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f964e924a84f749ba662d1d8abe98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:50.880823 2944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f964e924a84f749ba662d1d8abe98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:50.883466 3161 tablet_bootstrap.cc:492] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b: Bootstrap starting.
I20260927 04:06:50.883488 3162 tablet_bootstrap.cc:492] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Bootstrap starting.
I20260927 04:06:50.883860 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.875021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.17.184.195:43399)
0927 04:06:50.875550 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.875554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.875563 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:50.875599 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.875601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.875615 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.875687 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:50.875901 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.876732 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.878994 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.879770 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.879874 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.879926 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.883696 (+ 3770us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.883753 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":66,"threads_started":1}
I20260927 04:06:50.884186 3162 tablet_bootstrap.cc:654] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:50.884187 3161 tablet_bootstrap.cc:654] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:50.884517 3078 tablet_service.cc:1467] Processing CreateTablet for tablet a1f964e924a84f749ba662d1d8abe98f (DEFAULT_TABLE table=test-table [id=5c1834ca6a5b4cfcb0ec6c3e2762879e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:50.884717 3162 log.cc:824] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:50.884786 3161 log.cc:824] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:50.884867 3078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f964e924a84f749ba662d1d8abe98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:50.885445 3161 tablet_bootstrap.cc:492] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b: No bootstrap required, opened a new log
I20260927 04:06:50.885497 3162 tablet_bootstrap.cc:492] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: No bootstrap required, opened a new log
I20260927 04:06:50.885535 3161 ts_tablet_manager.cc:1397] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:50.885553 3162 ts_tablet_manager.cc:1397] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:06:50.887190 3161 raft_consensus.cc:348] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.887353 3161 raft_consensus.cc:374] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:50.887396 3161 raft_consensus.cc:729] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6db638c41ce84f9ab89688ba8a85762b, State: Initialized, Role: FOLLOWER
I20260927 04:06:50.887542 3161 consensus_queue.cc:260] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.887578 3162 raft_consensus.cc:348] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.887779 3162 raft_consensus.cc:374] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:50.887827 3162 raft_consensus.cc:729] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e42f87998b6f46c2b807f640e8df2a1d, State: Initialized, Role: FOLLOWER
I20260927 04:06:50.887846 3161 ts_tablet_manager.cc:1428] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 04:06:50.887951 2876 heartbeater.cc:503] Master 127.17.184.254:44095 was elected leader, sending a full tablet report...
I20260927 04:06:50.887943 3162 consensus_queue.cc:260] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.888206 3162 ts_tablet_manager.cc:1428] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:06:50.888273 3010 heartbeater.cc:503] Master 127.17.184.254:44095 was elected leader, sending a full tablet report...
I20260927 04:06:50.888718 3166 tablet_bootstrap.cc:492] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: Bootstrap starting.
I20260927 04:06:50.889436 3166 tablet_bootstrap.cc:654] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:50.889997 3166 log.cc:824] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:50.890748 3166 tablet_bootstrap.cc:492] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: No bootstrap required, opened a new log
I20260927 04:06:50.890821 3166 ts_tablet_manager.cc:1397] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:50.892174 3166 raft_consensus.cc:348] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.892354 3166 raft_consensus.cc:374] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:50.892400 3166 raft_consensus.cc:729] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe8694d6bfe49c7afe7e70f0d20a414, State: Initialized, Role: FOLLOWER
I20260927 04:06:50.892516 3166 consensus_queue.cc:260] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.892791 3166 ts_tablet_manager.cc:1428] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 04:06:50.892824 3144 heartbeater.cc:503] Master 127.17.184.254:44095 was elected leader, sending a full tablet report...
I20260927 04:06:50.928840 3165 raft_consensus.cc:482] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:50.928965 3165 raft_consensus.cc:504] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.929330 3167 raft_consensus.cc:482] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:50.929450 3165 leader_election.cc:288] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e42f87998b6f46c2b807f640e8df2a1d (127.17.184.194:34275), dbe8694d6bfe49c7afe7e70f0d20a414 (127.17.184.195:43399)
W20260927 04:06:50.929652 3011 tablet.cc:2369] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:06:50.929469 3167 raft_consensus.cc:504] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.930792 3167 leader_election.cc:288] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6db638c41ce84f9ab89688ba8a85762b (127.17.184.193:43285), dbe8694d6bfe49c7afe7e70f0d20a414 (127.17.184.195:43399)
I20260927 04:06:50.933641 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.929712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:39409 (local address 127.17.184.194:34275)
0927 04:06:50.929775 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.929778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.930013 (+ 235us) server_negotiation.cc:407] Connection header received
0927 04:06:50.930057 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.930059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.930071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.930138 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:50.930354 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.931186 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.932698 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.933249 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.933323 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.933333 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.933526 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.933573 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:06:50.934024 2878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.929594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:34275 (local address 127.17.184.193:39409)
0927 04:06:50.929826 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.929839 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.929894 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.930149 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.930151 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.930153 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:50.930257 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.930259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.931250 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.931251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.932599 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.932602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.933344 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.933346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.933439 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.933443 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.933506 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.933549 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":68}
I20260927 04:06:50.934068 3160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.930051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:53387 (local address 127.17.184.195:43399)
0927 04:06:50.930101 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.930104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.930112 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:06:50.930139 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.930140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.930149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.930212 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:50.930390 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.931107 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.932792 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.933328 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.933367 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.933374 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.933923 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.933957 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:06:50.934360 3170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.929651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:43399 (local address 127.17.184.193:53387)
0927 04:06:50.929827 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.929838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.929894 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.930224 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.930226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.930227 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:50.930290 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.930291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.931177 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.931179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.932599 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.932602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.933436 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.933436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.933514 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.933518 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.933565 (+ 47us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.933604 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":112,"thread_start_us":121,"threads_started":1}
I20260927 04:06:50.934361 2964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f964e924a84f749ba662d1d8abe98f" candidate_uuid: "6db638c41ce84f9ab89688ba8a85762b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42f87998b6f46c2b807f640e8df2a1d" is_pre_election: true
I20260927 04:06:50.934460 2964 raft_consensus.cc:2454] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6db638c41ce84f9ab89688ba8a85762b in term 0.
I20260927 04:06:50.934691 2766 leader_election.cc:302] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6db638c41ce84f9ab89688ba8a85762b, e42f87998b6f46c2b807f640e8df2a1d; no voters:
I20260927 04:06:50.934698 3098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f964e924a84f749ba662d1d8abe98f" candidate_uuid: "6db638c41ce84f9ab89688ba8a85762b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" is_pre_election: true
I20260927 04:06:50.934926 3165 raft_consensus.cc:2781] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:06:50.934988 3165 raft_consensus.cc:482] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:06:50.935015 3165 raft_consensus.cc:3037] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:50.935602 3098 raft_consensus.cc:2454] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6db638c41ce84f9ab89688ba8a85762b in term 0.
I20260927 04:06:50.935745 3012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.930623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:43399 (local address 127.17.184.194:58091)
0927 04:06:50.930977 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.930987 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.931029 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.931472 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.931472 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.931474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:50.931518 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.931519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.932520 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.932520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.933677 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.933680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.935487 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.935488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.935567 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.935571 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.935631 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.935669 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":17,"thread_start_us":108,"threads_started":1}
I20260927 04:06:50.935885 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.930939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:37749 (local address 127.17.184.193:43285)
0927 04:06:50.931106 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.931108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.931115 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:06:50.931136 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.931136 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.931145 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.931194 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:50.931412 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.932234 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.934863 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.935483 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.935527 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.935534 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.935767 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.935808 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260927 04:06:50.935916 3165 raft_consensus.cc:504] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.936051 3165 leader_election.cc:288] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [CANDIDATE]: Term 1 election: Requested vote from peers e42f87998b6f46c2b807f640e8df2a1d (127.17.184.194:34275), dbe8694d6bfe49c7afe7e70f0d20a414 (127.17.184.195:43399)
I20260927 04:06:50.936089 3172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.931043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:58091 (local address 127.17.184.195:43399)
0927 04:06:50.931320 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.931322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.931329 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:06:50.931350 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.931351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.931362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.931583 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:50.931615 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.932427 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.933773 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.935391 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.935474 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.935482 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.935954 (+ 472us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.936004 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":86,"threads_started":1}
I20260927 04:06:50.936128 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.930614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:43285 (local address 127.17.184.194:37749)
0927 04:06:50.930781 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.930791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:50.930834 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:50.931240 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:50.931241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:50.931243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:50.931315 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.931317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.932335 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.932335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.934755 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:50.934758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.935577 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.935578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:50.935659 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.935663 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:50.935994 (+ 331us) client_negotiation.cc:240] Negotiation successful
0927 04:06:50.936038 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":81}
I20260927 04:06:50.936285 2964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f964e924a84f749ba662d1d8abe98f" candidate_uuid: "6db638c41ce84f9ab89688ba8a85762b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e42f87998b6f46c2b807f640e8df2a1d"
I20260927 04:06:50.936331 3098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f964e924a84f749ba662d1d8abe98f" candidate_uuid: "6db638c41ce84f9ab89688ba8a85762b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414"
I20260927 04:06:50.936362 2964 raft_consensus.cc:3037] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:50.936364 3097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f964e924a84f749ba662d1d8abe98f" candidate_uuid: "e42f87998b6f46c2b807f640e8df2a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" is_pre_election: true
I20260927 04:06:50.936410 3098 raft_consensus.cc:3037] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:50.936805 2830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1f964e924a84f749ba662d1d8abe98f" candidate_uuid: "e42f87998b6f46c2b807f640e8df2a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6db638c41ce84f9ab89688ba8a85762b" is_pre_election: true
I20260927 04:06:50.936920 2830 raft_consensus.cc:2379] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e42f87998b6f46c2b807f640e8df2a1d in current term 1: Already voted for candidate 6db638c41ce84f9ab89688ba8a85762b in this term.
I20260927 04:06:50.937112 2897 leader_election.cc:302] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e42f87998b6f46c2b807f640e8df2a1d; no voters: 6db638c41ce84f9ab89688ba8a85762b, dbe8694d6bfe49c7afe7e70f0d20a414
I20260927 04:06:50.937302 2964 raft_consensus.cc:2454] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6db638c41ce84f9ab89688ba8a85762b in term 1.
I20260927 04:06:50.937302 3098 raft_consensus.cc:2454] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6db638c41ce84f9ab89688ba8a85762b in term 1.
I20260927 04:06:50.937408 3167 raft_consensus.cc:2726] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 04:06:50.937472 2766 leader_election.cc:302] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6db638c41ce84f9ab89688ba8a85762b, e42f87998b6f46c2b807f640e8df2a1d; no voters:
I20260927 04:06:50.937556 3165 raft_consensus.cc:2781] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:50.937702 3165 raft_consensus.cc:686] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 1 LEADER]: Becoming Leader. State: Replica: 6db638c41ce84f9ab89688ba8a85762b, State: Running, Role: LEADER
I20260927 04:06:50.937770 3165 consensus_queue.cc:237] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } }
I20260927 04:06:50.938582 2684 catalog_manager.cc:5495] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6db638c41ce84f9ab89688ba8a85762b (127.17.184.193). New cstate: current_term: 1 leader_uuid: "6db638c41ce84f9ab89688ba8a85762b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6db638c41ce84f9ab89688ba8a85762b" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 } health_report { overall_health: UNKNOWN } } }
I20260927 04:06:50.953164 3159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.950096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.17.184.193:43285)
0927 04:06:50.950155 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:50.950156 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:50.950165 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:50.950202 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:50.950202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:50.950218 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:50.950298 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:06:50.950434 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.951178 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.952165 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:50.952471 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:50.952541 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:50.952742 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:06:50.952880 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:06:50.952969 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:50.953020 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 04:06:50.953043 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 04:06:50.955703 3098 raft_consensus.cc:1261] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 1 FOLLOWER]: Refusing update from remote peer 6db638c41ce84f9ab89688ba8a85762b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:50.955704 2964 raft_consensus.cc:1261] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 1 FOLLOWER]: Refusing update from remote peer 6db638c41ce84f9ab89688ba8a85762b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:50.956027 3165 consensus_queue.cc:1035] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e42f87998b6f46c2b807f640e8df2a1d" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 34275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:50.956164 3173 consensus_queue.cc:1035] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbe8694d6bfe49c7afe7e70f0d20a414" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:50.958482 3176 mvcc.cc:205] Tried to move back new op lower bound from 7333814316871610368 to 7333814316801634304. Current Snapshot: MvccSnapshot[applied={T|T < 7333814316871610368}]
I20260927 04:06:50.959306 3178 mvcc.cc:205] Tried to move back new op lower bound from 7333814316871610368 to 7333814316801634304. Current Snapshot: MvccSnapshot[applied={T|T < 7333814316871610368 or (T in {7333814316871610368})}]
I20260927 04:06:50.959488 3177 mvcc.cc:205] Tried to move back new op lower bound from 7333814316871610368 to 7333814316801634304. Current Snapshot: MvccSnapshot[applied={T|T < 7333814316871610368}]
I20260927 04:06:50.960819 2684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:50.960930 2684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:50.961659 2684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c42241e1934a4aef948eba3255234319: Sending DeleteTablet for 3 replicas of tablet a1f964e924a84f749ba662d1d8abe98f
I20260927 04:06:50.962028 2810 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f964e924a84f749ba662d1d8abe98f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:50 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:50192
I20260927 04:06:50.962030 2944 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f964e924a84f749ba662d1d8abe98f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:50 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:41952
I20260927 04:06:50.962131 3078 tablet_service.cc:1514] Processing DeleteTablet for tablet a1f964e924a84f749ba662d1d8abe98f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:50 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:36340
I20260927 04:06:50.962261 3187 tablet_replica.cc:330] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b: stopping tablet replica
I20260927 04:06:50.962361 3187 raft_consensus.cc:2229] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:06:50.962488 3187 raft_consensus.cc:2258] T a1f964e924a84f749ba662d1d8abe98f P 6db638c41ce84f9ab89688ba8a85762b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:50.962747 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 2747
I20260927 04:06:50.963402 3188 tablet_replica.cc:330] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: stopping tablet replica
I20260927 04:06:50.963511 3188 raft_consensus.cc:2229] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:50.963618 3188 raft_consensus.cc:2258] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:50.963639 3186 tablet_replica.cc:330] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: stopping tablet replica
I20260927 04:06:50.963737 3186 raft_consensus.cc:2229] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:50.963894 3186 raft_consensus.cc:2258] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:50.965531 3188 ts_tablet_manager.cc:1905] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:50.965585 3186 ts_tablet_manager.cc:1905] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:50.967424 3188 ts_tablet_manager.cc:1918] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:50.967550 3188 log.cc:1196] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/a1f964e924a84f749ba662d1d8abe98f
I20260927 04:06:50.967545 3186 ts_tablet_manager.cc:1918] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:50.967617 3186 log.cc:1196] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/a1f964e924a84f749ba662d1d8abe98f
I20260927 04:06:50.967840 3188 ts_tablet_manager.cc:1939] T a1f964e924a84f749ba662d1d8abe98f P dbe8694d6bfe49c7afe7e70f0d20a414: Deleting consensus metadata
I20260927 04:06:50.967900 3186 ts_tablet_manager.cc:1939] T a1f964e924a84f749ba662d1d8abe98f P e42f87998b6f46c2b807f640e8df2a1d: Deleting consensus metadata
I20260927 04:06:50.968339 2670 catalog_manager.cc:4841] TS dbe8694d6bfe49c7afe7e70f0d20a414 (127.17.184.195:43399): tablet a1f964e924a84f749ba662d1d8abe98f (table test-table [id=5c1834ca6a5b4cfcb0ec6c3e2762879e]) successfully deleted
I20260927 04:06:50.968637 2672 catalog_manager.cc:4841] TS e42f87998b6f46c2b807f640e8df2a1d (127.17.184.194:34275): tablet a1f964e924a84f749ba662d1d8abe98f (table test-table [id=5c1834ca6a5b4cfcb0ec6c3e2762879e]) successfully deleted
W20260927 04:06:50.969640 2669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:43285 (error 108)
I20260927 04:06:50.969951 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 2881
I20260927 04:06:50.970274 3156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:50.970126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:43285 (local address 127.0.0.1:50202)
0927 04:06:50.970185 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:50.970229 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:43285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 04:06:50.970394 2669 catalog_manager.cc:4584] TS 6db638c41ce84f9ab89688ba8a85762b (127.17.184.193:43285): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a1f964e924a84f749ba662d1d8abe98f: Network error: Client connection negotiation failed: client connection to 127.17.184.193:43285: connect: Connection refused (error 111)
I20260927 04:06:50.975117 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 3015
I20260927 04:06:50.980160 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 2653
2026-09-27T04:06:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2036 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T04:06:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:06:51Z Disabled control of system clock
I20260927 04:06:51.009306 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41415
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:41415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:51.090996 3193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:51.091233 3193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:51.091254 3193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:51.092507 3193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:51.092558 3193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:51.092571 3193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:51.092584 3193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:51.092597 3193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:51.094087 3193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:41415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:41415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:51.094368 3193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:51.094715 3193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:51.099800 3198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:51.099944 3193 server_base.cc:1034] running on GCE node
W20260927 04:06:51.099800 3199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:51.100069 3201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:51.100365 3193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:51.100663 3193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:51.101825 3193 hybrid_clock.cc:648] HybridClock initialized: now 1790482011101815 us; error 49 us; skew 500 ppm
I20260927 04:06:51.103205 3193 webserver.cc:460] Webserver started at http://127.17.184.254:40017/ using document root <none> and password file <none>
I20260927 04:06:51.103459 3193 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:51.103530 3193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:51.103700 3193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:51.104799 3193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "0a792c651dfa4449839094cae388cd1f"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "f09c78eae8c242f40e61cba6136b4624"
server_key_iv: "84e08c004b596e1b731a12c81c79598b"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.105178 3193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "0a792c651dfa4449839094cae388cd1f"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "f09c78eae8c242f40e61cba6136b4624"
server_key_iv: "84e08c004b596e1b731a12c81c79598b"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.106489 3193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:51.107327 3207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.107496 3193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.107595 3193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "0a792c651dfa4449839094cae388cd1f"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "f09c78eae8c242f40e61cba6136b4624"
server_key_iv: "84e08c004b596e1b731a12c81c79598b"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.107677 3193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:51.115914 3193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:51.116266 3193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:51.116425 3193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:51.120455 3193 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:41415
I20260927 04:06:51.120489 3259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:41415 every 8 connection(s)
I20260927 04:06:51.120834 3193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:51.121905 3260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:51.124020 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 3193
I20260927 04:06:51.124135 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:06:51.124331 18147 external_mini_cluster.cc:1428] Setting key dab652c0c2e868de244be18c39416c0e
I20260927 04:06:51.124724 3260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f: Bootstrap starting.
I20260927 04:06:51.125370 3260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:51.125717 3260 log.cc:824] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:51.126433 3260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f: No bootstrap required, opened a new log
I20260927 04:06:51.128196 3260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a792c651dfa4449839094cae388cd1f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41415 } }
I20260927 04:06:51.128352 3260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:51.128413 3260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a792c651dfa4449839094cae388cd1f, State: Initialized, Role: FOLLOWER
I20260927 04:06:51.128628 3260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a792c651dfa4449839094cae388cd1f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41415 } }
I20260927 04:06:51.128731 3265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:51.128778 3265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a792c651dfa4449839094cae388cd1f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41415 } }
I20260927 04:06:51.128717 3260 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:51.128871 3265 leader_election.cc:302] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a792c651dfa4449839094cae388cd1f; no voters:
I20260927 04:06:51.128944 3260 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:51.128974 3260 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:51.129067 3265 leader_election.cc:288] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260927 04:06:51.129562 3260 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a792c651dfa4449839094cae388cd1f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41415 } }
I20260927 04:06:51.129652 3260 leader_election.cc:302] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a792c651dfa4449839094cae388cd1f; no voters:
I20260927 04:06:51.129745 3265 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:51.129731 3260 leader_election.cc:288] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:06:51.129878 3265 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 1 LEADER]: Becoming Leader. State: Replica: 0a792c651dfa4449839094cae388cd1f, State: Running, Role: LEADER
I20260927 04:06:51.130034 3260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:51.130007 3265 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a792c651dfa4449839094cae388cd1f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41415 } }
I20260927 04:06:51.130084 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.124906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52476 (local address 127.17.184.254:41415)
0927 04:06:51.125100 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.125108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.125138 (+ 30us) server_negotiation.cc:407] Connection header received
0927 04:06:51.125639 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.125646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.125716 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.125869 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:51.126095 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.127084 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.127861 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.128273 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.128407 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.129239 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:51.129242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:51.129243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:51.129428 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:51.129460 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:51.129470 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:51.129555 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.129630 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.129706 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260927 04:06:51.131348 3266 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [term 1 LEADER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 04:06:51.131595 3267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a792c651dfa4449839094cae388cd1f. Latest consensus state: current_term: 1 leader_uuid: "0a792c651dfa4449839094cae388cd1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a792c651dfa4449839094cae388cd1f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41415 } } }
I20260927 04:06:51.131644 3268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a792c651dfa4449839094cae388cd1f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a792c651dfa4449839094cae388cd1f" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 41415 } } }
I20260927 04:06:51.131682 3267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:51.131688 3268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f [sys.catalog]: This master's current role is: LEADER
W20260927 04:06:51.132193 3281 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:06:51.132258 3281 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:06:51.132347 3283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:51.132534 3283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:06:51.134011 3283 catalog_manager.cc:1347] Generated new cluster ID: bae9e99ccf574b0c96cc5b43ebca6688
I20260927 04:06:51.134061 3283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:06:51.138875 3283 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:06:51.139393 3283 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:51.141160 3283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f: Generated new TSK 0
I20260927 04:06:51.141322 3283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:06:51.143276 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41415
--builtin_ntp_servers=127.17.184.212:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:51.225514 3285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:51.225770 3285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:51.225811 3285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:51.227116 3285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:51.227176 3285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:51.227221 3285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:06:51.228714 3285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:51.228987 3285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:51.229295 3285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:51.235571 3290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:51.235687 3293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:51.235575 3291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:51.235957 3285 server_base.cc:1034] running on GCE node
I20260927 04:06:51.236258 3285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:51.236521 3285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:51.237699 3285 hybrid_clock.cc:648] HybridClock initialized: now 1790482011237675 us; error 37 us; skew 500 ppm
I20260927 04:06:51.239276 3285 webserver.cc:460] Webserver started at http://127.17.184.193:40101/ using document root <none> and password file <none>
I20260927 04:06:51.239550 3285 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:51.239614 3285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:51.239744 3285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:51.241230 3285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "399fb816f71e423e96dea2525a9950aa"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "f0b63a3fd62cdd22de699160c856a2b0"
server_key_iv: "3db93ea6bad6db93a48d1ec1f4ad24d9"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.241650 3285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "399fb816f71e423e96dea2525a9950aa"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "f0b63a3fd62cdd22de699160c856a2b0"
server_key_iv: "3db93ea6bad6db93a48d1ec1f4ad24d9"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.243363 3285 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260927 04:06:51.244201 3299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.244400 3285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:06:51.244488 3285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "399fb816f71e423e96dea2525a9950aa"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "f0b63a3fd62cdd22de699160c856a2b0"
server_key_iv: "3db93ea6bad6db93a48d1ec1f4ad24d9"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.244558 3285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:51.262642 3285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:51.263069 3285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:51.263226 3285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:51.263484 3285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:51.263878 3285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:51.263921 3285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.263949 3285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:51.263965 3285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.270478 3285 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:33469
I20260927 04:06:51.270555 3412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:33469 every 8 connection(s)
I20260927 04:06:51.270964 3285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:06:51.276757 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.271810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:49791 (local address 127.17.184.254:41415)
0927 04:06:51.271881 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.271883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.272137 (+ 254us) server_negotiation.cc:407] Connection header received
0927 04:06:51.272636 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.272637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.272648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.272713 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:51.273405 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.274112 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.275240 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.275426 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.275489 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.276447 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:51.276448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:51.276449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:51.276453 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:51.276468 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:51.276472 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:51.276528 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.276672 (+ 144us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.276693 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 04:06:51.276860 3415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.271801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41415 (local address 127.17.184.193:49791)
0927 04:06:51.271996 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.272039 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:51.272534 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:51.272789 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:51.272792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:51.272800 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:51.273300 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.273303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.274242 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.274243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.275134 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.275138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.275528 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.275529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.275647 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.276294 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:51.276294 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:51.276323 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:51.276347 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:51.276542 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:51.276546 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:51.276549 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:51.276610 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 04:06:51.276692 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260927 04:06:51.277462 3413 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41415
I20260927 04:06:51.277560 3413 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:51.277710 3413 heartbeater.cc:511] Master 127.17.184.254:41415 requested a full tablet report, sending...
I20260927 04:06:51.278239 3224 ts_manager.cc:194] Registered new tserver with Master: 399fb816f71e423e96dea2525a9950aa (127.17.184.193:33469)
I20260927 04:06:51.278528 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 3285
I20260927 04:06:51.278636 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:06:51.278841 18147 external_mini_cluster.cc:1428] Setting key da9c1015fc06f708f443bb4ae27c889a
I20260927 04:06:51.279212 3224 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.193:49791
I20260927 04:06:51.280054 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41415
--builtin_ntp_servers=127.17.184.212:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:51.361105 3416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:51.361299 3416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:51.361320 3416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:51.362448 3416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:51.362486 3416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:51.362510 3416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:06:51.363902 3416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:51.364146 3416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:51.364456 3416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:06:51.370261 3416 server_base.cc:1034] running on GCE node
W20260927 04:06:51.370246 3422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:51.370237 3424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:51.370246 3421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:51.370731 3416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:51.371042 3416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:51.372211 3416 hybrid_clock.cc:648] HybridClock initialized: now 1790482011372191 us; error 31 us; skew 500 ppm
I20260927 04:06:51.373526 3416 webserver.cc:460] Webserver started at http://127.17.184.194:46417/ using document root <none> and password file <none>
I20260927 04:06:51.373780 3416 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:51.373828 3416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:51.374017 3416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:51.375347 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "3f8528f7440a4e5b9659a996f49051c5"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "3332ed8a425f865914c0d137d2f3c454"
server_key_iv: "97d20da4f1b03c6d5ec944a7c4bbd9e1"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.375725 3416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "3f8528f7440a4e5b9659a996f49051c5"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "3332ed8a425f865914c0d137d2f3c454"
server_key_iv: "97d20da4f1b03c6d5ec944a7c4bbd9e1"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.377090 3416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:51.377892 3430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.378073 3416 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:06:51.378149 3416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "3f8528f7440a4e5b9659a996f49051c5"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "3332ed8a425f865914c0d137d2f3c454"
server_key_iv: "97d20da4f1b03c6d5ec944a7c4bbd9e1"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.378248 3416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:51.401811 3416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:51.402168 3416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:51.402289 3416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:51.402608 3416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:51.403017 3416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:51.403074 3416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.403100 3416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:51.403153 3416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.409360 3416 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:45413
I20260927 04:06:51.409406 3543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:45413 every 8 connection(s)
I20260927 04:06:51.409869 3416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:06:51.414974 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 3416
I20260927 04:06:51.415102 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:06:51.415293 18147 external_mini_cluster.cc:1428] Setting key 1918c7a06875ac733eeafb1df8d9ee7e
I20260927 04:06:51.415705 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.410659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:37077 (local address 127.17.184.254:41415)
0927 04:06:51.410757 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.410759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.411030 (+ 271us) server_negotiation.cc:407] Connection header received
0927 04:06:51.411553 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.411554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.411565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.411613 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:51.412330 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.412846 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.413926 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.414150 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.414227 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.415351 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:51.415352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:51.415353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:51.415358 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:51.415374 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:51.415379 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:51.415458 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.415589 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.415618 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 04:06:51.415957 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.410615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41415 (local address 127.17.184.194:37077)
0927 04:06:51.410846 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.410883 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:51.411453 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:51.411678 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:51.411681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:51.411689 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:51.412236 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.412240 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.412944 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.412945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.413786 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.413789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.414256 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.414257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.414374 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.415164 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:51.415165 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:51.415209 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:51.415239 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:51.415458 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:51.415465 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:51.415469 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:51.415588 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 04:06:51.415701 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":117,"threads_started":1}
I20260927 04:06:51.416299 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41415
--builtin_ntp_servers=127.17.184.212:43673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 04:06:51.416494 3544 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41415
I20260927 04:06:51.416599 3544 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:51.416786 3544 heartbeater.cc:511] Master 127.17.184.254:41415 requested a full tablet report, sending...
I20260927 04:06:51.417186 3223 ts_manager.cc:194] Registered new tserver with Master: 3f8528f7440a4e5b9659a996f49051c5 (127.17.184.194:45413)
I20260927 04:06:51.417696 3223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.194:37077
W20260927 04:06:51.499609 3547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:51.499832 3547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:51.499866 3547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:51.500976 3547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:51.501044 3547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:51.501087 3547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:06:51.502408 3547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:41415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:51.502681 3547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:51.502991 3547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:51.507965 3555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:51.507959 3553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:51.508060 3552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:51.508253 3547 server_base.cc:1034] running on GCE node
I20260927 04:06:51.508535 3547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:51.508755 3547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:51.509958 3547 hybrid_clock.cc:648] HybridClock initialized: now 1790482011509942 us; error 29 us; skew 500 ppm
I20260927 04:06:51.511533 3547 webserver.cc:460] Webserver started at http://127.17.184.195:40043/ using document root <none> and password file <none>
I20260927 04:06:51.511816 3547 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:51.511868 3547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:51.512043 3547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:51.513151 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "9a8f958b3bf642bd84ca0d730b0da236"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "dd1fdd3ecbb66f225684d5f5bedf0da8"
server_key_iv: "2ff887abda0220302d6b4700b5386a6d"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.513527 3547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "9a8f958b3bf642bd84ca0d730b0da236"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "dd1fdd3ecbb66f225684d5f5bedf0da8"
server_key_iv: "2ff887abda0220302d6b4700b5386a6d"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.514947 3547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:51.515717 3561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.515902 3547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:06:51.516013 3547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "9a8f958b3bf642bd84ca0d730b0da236"
format_stamp: "Formatted at 2026-09-27 04:06:51 on dist-test-slave-zqrn"
server_key: "dd1fdd3ecbb66f225684d5f5bedf0da8"
server_key_iv: "2ff887abda0220302d6b4700b5386a6d"
server_key_version: "encryptionkey@0"
I20260927 04:06:51.516108 3547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:51.525578 3547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:51.525884 3547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:51.526049 3547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:51.526281 3547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:51.526639 3547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:51.526702 3547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.526748 3547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:51.526779 3547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:51.533497 3547 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:35171
I20260927 04:06:51.533540 3674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:35171 every 8 connection(s)
I20260927 04:06:51.533936 3547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:06:51.539633 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.534726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:45085 (local address 127.17.184.254:41415)
0927 04:06:51.534813 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.534816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.535101 (+ 285us) server_negotiation.cc:407] Connection header received
0927 04:06:51.535645 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.535646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.535657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.535708 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:51.536511 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.536996 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.538019 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.538227 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.538295 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.539373 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:51.539374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:51.539374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:51.539379 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:51.539394 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:51.539398 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:51.539451 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.539548 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.539573 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 04:06:51.539862 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.534697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:41415 (local address 127.17.184.195:45085)
0927 04:06:51.534950 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.534997 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:51.535514 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:51.535779 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:51.535782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:51.535790 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:51.536423 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.536433 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.537082 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.537083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.537914 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.537916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.538340 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.538341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.538479 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.539228 (+ 749us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:51.539228 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:51.539257 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:51.539280 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:51.539451 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:51.539453 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:51.539456 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:51.539545 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 04:06:51.539624 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
I20260927 04:06:51.540482 3675 heartbeater.cc:347] Connected to a master server at 127.17.184.254:41415
I20260927 04:06:51.540578 3675 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:51.540737 3675 heartbeater.cc:511] Master 127.17.184.254:41415 requested a full tablet report, sending...
I20260927 04:06:51.541100 3223 ts_manager.cc:194] Registered new tserver with Master: 9a8f958b3bf642bd84ca0d730b0da236 (127.17.184.195:35171)
I20260927 04:06:51.541438 3223 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.195:45085
I20260927 04:06:51.541819 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 3547
I20260927 04:06:51.541953 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:06:51.542164 18147 external_mini_cluster.cc:1428] Setting key f735f714e19c45087caeffdf94f52782
I20260927 04:06:51.543045 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:06:51.547111 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.544370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.17.184.254:41415)
0927 04:06:51.544408 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.544410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.544540 (+ 130us) server_negotiation.cc:407] Connection header received
0927 04:06:51.544594 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.544595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.544602 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.544644 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:51.544787 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.545254 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.546035 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.546221 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.546271 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.546882 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:51.546883 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:51.546883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:51.546887 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:51.546918 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:51.546923 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:51.546960 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.547039 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.547059 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:06:51.548830 3223 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:06:51.550310 3223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:06:51.556164 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.552372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:35171 (local address 127.0.0.1:45818)
0927 04:06:51.552742 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.552753 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:51.552807 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:51.553058 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:51.553059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:51.553060 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:51.553121 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.553123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.554024 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.554025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.554951 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.554953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.555842 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.555843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.555924 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.555939 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 04:06:51.556031 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 04:06:51.556085 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":97,"threads_started":1}
I20260927 04:06:51.556167 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.552629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45818 (local address 127.17.184.195:35171)
0927 04:06:51.552803 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.552806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.552817 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:06:51.552912 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.552914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.552931 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.552998 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:51.553200 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.553937 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.555061 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.555761 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.555844 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.555930 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.556044 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.556080 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260927 04:06:51.557024 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.552594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33469 (local address 127.0.0.1:60072)
0927 04:06:51.552870 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.552880 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:51.552920 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:51.553246 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:51.553246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:51.553249 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:51.553290 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.553291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.554697 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.554698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.555827 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.555830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.556784 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.556785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.556847 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.556850 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:51.556922 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 04:06:51.556960 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":84,"threads_started":1}
I20260927 04:06:51.557060 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.552815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60072 (local address 127.17.184.193:33469)
0927 04:06:51.553019 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.553023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.553035 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:06:51.553075 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.553077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.553095 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.553176 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:51.553372 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.554609 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.555915 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.556551 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.556645 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.556736 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.556937 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.556981 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260927 04:06:51.557163 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.552448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54402 (local address 127.17.184.194:45413)
0927 04:06:51.552615 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.552620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.552631 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:06:51.552765 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.552769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.552789 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.552851 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:51.553184 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.554485 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.555769 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.556621 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.556694 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.556812 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.557022 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.557070 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260927 04:06:51.557228 3609 tablet_service.cc:1467] Processing CreateTablet for tablet bc33ebcd6ca6450dba7ebcf35896eef3 (DEFAULT_TABLE table=test-table [id=a0baab34c97c470ba11ee5d38788057c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:51.557509 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.552359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:45413 (local address 127.0.0.1:54402)
0927 04:06:51.552524 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.552567 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:51.552635 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:51.552966 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:51.552970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:51.552973 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:51.553057 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.553059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.554597 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.554598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.555673 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.555676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.556852 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.556853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.556920 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.556925 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:06:51.557264 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 04:06:51.557427 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":22,"thread_start_us":52,"threads_started":1}
I20260927 04:06:51.557618 3609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc33ebcd6ca6450dba7ebcf35896eef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:51.557825 3347 tablet_service.cc:1467] Processing CreateTablet for tablet bc33ebcd6ca6450dba7ebcf35896eef3 (DEFAULT_TABLE table=test-table [id=a0baab34c97c470ba11ee5d38788057c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:51.558146 3347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc33ebcd6ca6450dba7ebcf35896eef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:51.558202 3478 tablet_service.cc:1467] Processing CreateTablet for tablet bc33ebcd6ca6450dba7ebcf35896eef3 (DEFAULT_TABLE table=test-table [id=a0baab34c97c470ba11ee5d38788057c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:51.558537 3478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc33ebcd6ca6450dba7ebcf35896eef3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:51.560674 3689 tablet_bootstrap.cc:492] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa: Bootstrap starting.
I20260927 04:06:51.561298 3690 tablet_bootstrap.cc:492] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: Bootstrap starting.
I20260927 04:06:51.561340 3689 tablet_bootstrap.cc:654] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:51.561612 3689 log.cc:824] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:51.561960 3690 tablet_bootstrap.cc:654] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:51.562219 3692 tablet_bootstrap.cc:492] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: Bootstrap starting.
I20260927 04:06:51.562338 3689 tablet_bootstrap.cc:492] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa: No bootstrap required, opened a new log
I20260927 04:06:51.562398 3689 ts_tablet_manager.cc:1397] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:51.562422 3690 log.cc:824] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:51.562858 3692 tablet_bootstrap.cc:654] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:51.563138 3690 tablet_bootstrap.cc:492] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: No bootstrap required, opened a new log
I20260927 04:06:51.563184 3690 ts_tablet_manager.cc:1397] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:06:51.563223 3692 log.cc:824] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:51.564057 3692 tablet_bootstrap.cc:492] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: No bootstrap required, opened a new log
I20260927 04:06:51.564131 3692 ts_tablet_manager.cc:1397] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:06:51.564634 3689 raft_consensus.cc:348] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.564759 3689 raft_consensus.cc:374] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:51.564779 3689 raft_consensus.cc:729] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399fb816f71e423e96dea2525a9950aa, State: Initialized, Role: FOLLOWER
I20260927 04:06:51.564898 3689 consensus_queue.cc:260] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.565147 3689 ts_tablet_manager.cc:1428] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:06:51.565194 3413 heartbeater.cc:503] Master 127.17.184.254:41415 was elected leader, sending a full tablet report...
I20260927 04:06:51.566244 3690 raft_consensus.cc:348] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.566417 3690 raft_consensus.cc:374] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:51.566457 3690 raft_consensus.cc:729] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8f958b3bf642bd84ca0d730b0da236, State: Initialized, Role: FOLLOWER
I20260927 04:06:51.566584 3690 consensus_queue.cc:260] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.566856 3690 ts_tablet_manager.cc:1428] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260927 04:06:51.566882 3675 heartbeater.cc:503] Master 127.17.184.254:41415 was elected leader, sending a full tablet report...
I20260927 04:06:51.566977 3692 raft_consensus.cc:348] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.567173 3692 raft_consensus.cc:374] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:51.567226 3692 raft_consensus.cc:729] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8528f7440a4e5b9659a996f49051c5, State: Initialized, Role: FOLLOWER
I20260927 04:06:51.567354 3692 consensus_queue.cc:260] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.567668 3692 ts_tablet_manager.cc:1428] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260927 04:06:51.567740 3544 heartbeater.cc:503] Master 127.17.184.254:41415 was elected leader, sending a full tablet report...
W20260927 04:06:51.660540 3545 tablet.cc:2369] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:06:51.686648 3696 raft_consensus.cc:482] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:51.686784 3696 raft_consensus.cc:504] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.687287 3696 leader_election.cc:288] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f8528f7440a4e5b9659a996f49051c5 (127.17.184.194:45413), 399fb816f71e423e96dea2525a9950aa (127.17.184.193:33469)
I20260927 04:06:51.690209 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.687599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:35953 (local address 127.17.184.194:45413)
0927 04:06:51.687641 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.687643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.687656 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:06:51.687690 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.687691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.687701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.687794 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:51.687987 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.688455 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.689473 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.689841 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.689933 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.689942 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.690103 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.690141 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:06:51.690209 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.687487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:45413 (local address 127.17.184.195:35953)
0927 04:06:51.687565 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.687580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:51.687633 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:51.687806 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:51.687807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:51.687808 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:51.687890 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.687891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.688612 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.688613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.689362 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.689364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.689926 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.689928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.690001 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.690004 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:51.690097 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 04:06:51.690139 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 04:06:51.690519 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.687644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:44583 (local address 127.17.184.193:33469)
0927 04:06:51.687693 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.687695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.687830 (+ 135us) server_negotiation.cc:407] Connection header received
0927 04:06:51.687926 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.687926 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.687946 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.688016 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:51.688194 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.688720 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.689736 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.690123 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.690213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.690220 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.690411 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.690465 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:06:51.690547 3498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc33ebcd6ca6450dba7ebcf35896eef3" candidate_uuid: "9a8f958b3bf642bd84ca0d730b0da236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8528f7440a4e5b9659a996f49051c5" is_pre_election: true
I20260927 04:06:51.690723 3498 raft_consensus.cc:2454] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a8f958b3bf642bd84ca0d730b0da236 in term 0.
I20260927 04:06:51.690765 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.687509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33469 (local address 127.17.184.195:44583)
0927 04:06:51.687751 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.687760 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:51.687846 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:51.688021 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:51.688022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:51.688023 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:51.688113 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.688114 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.688813 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.688814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.689625 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:51.689627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.690213 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.690214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:51.690301 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.690306 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:06:51.690639 (+ 333us) client_negotiation.cc:240] Negotiation successful
0927 04:06:51.690688 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
I20260927 04:06:51.690996 3564 leader_election.cc:302] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8528f7440a4e5b9659a996f49051c5, 9a8f958b3bf642bd84ca0d730b0da236; no voters:
I20260927 04:06:51.691152 3696 raft_consensus.cc:2781] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:06:51.691159 3367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc33ebcd6ca6450dba7ebcf35896eef3" candidate_uuid: "9a8f958b3bf642bd84ca0d730b0da236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399fb816f71e423e96dea2525a9950aa" is_pre_election: true
I20260927 04:06:51.691249 3696 raft_consensus.cc:482] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:06:51.691285 3696 raft_consensus.cc:3037] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:51.691310 3367 raft_consensus.cc:2454] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a8f958b3bf642bd84ca0d730b0da236 in term 0.
I20260927 04:06:51.692103 3696 raft_consensus.cc:504] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.692271 3696 leader_election.cc:288] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [CANDIDATE]: Term 1 election: Requested vote from peers 3f8528f7440a4e5b9659a996f49051c5 (127.17.184.194:45413), 399fb816f71e423e96dea2525a9950aa (127.17.184.193:33469)
I20260927 04:06:51.692446 3367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc33ebcd6ca6450dba7ebcf35896eef3" candidate_uuid: "9a8f958b3bf642bd84ca0d730b0da236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399fb816f71e423e96dea2525a9950aa"
I20260927 04:06:51.692466 3498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc33ebcd6ca6450dba7ebcf35896eef3" candidate_uuid: "9a8f958b3bf642bd84ca0d730b0da236" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f8528f7440a4e5b9659a996f49051c5"
I20260927 04:06:51.692533 3367 raft_consensus.cc:3037] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:51.692552 3498 raft_consensus.cc:3037] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:51.693473 3367 raft_consensus.cc:2454] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a8f958b3bf642bd84ca0d730b0da236 in term 1.
I20260927 04:06:51.693475 3498 raft_consensus.cc:2454] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a8f958b3bf642bd84ca0d730b0da236 in term 1.
I20260927 04:06:51.693642 3564 leader_election.cc:302] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8528f7440a4e5b9659a996f49051c5, 9a8f958b3bf642bd84ca0d730b0da236; no voters:
I20260927 04:06:51.693756 3696 raft_consensus.cc:2781] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:51.693940 3696 raft_consensus.cc:686] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 1 LEADER]: Becoming Leader. State: Replica: 9a8f958b3bf642bd84ca0d730b0da236, State: Running, Role: LEADER
I20260927 04:06:51.694051 3696 consensus_queue.cc:237] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } }
I20260927 04:06:51.694885 3223 catalog_manager.cc:5495] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a8f958b3bf642bd84ca0d730b0da236 (127.17.184.195). New cstate: current_term: 1 leader_uuid: "9a8f958b3bf642bd84ca0d730b0da236" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a8f958b3bf642bd84ca0d730b0da236" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 35171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 } health_report { overall_health: UNKNOWN } } }
I20260927 04:06:51.701269 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.699065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.17.184.195:35171)
0927 04:06:51.699130 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:51.699132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:51.699140 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:06:51.699185 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:51.699185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:51.699199 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:51.699256 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:06:51.699346 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.699801 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.700539 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:51.700754 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:51.700819 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:51.700954 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:06:51.701042 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:06:51.701108 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:51.701126 (+ 18us) server_negotiation.cc:299] Negotiation successful
0927 04:06:51.701141 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:06:51.703001 3367 raft_consensus.cc:1261] T bc33ebcd6ca6450dba7ebcf35896eef3 P 399fb816f71e423e96dea2525a9950aa [term 1 FOLLOWER]: Refusing update from remote peer 9a8f958b3bf642bd84ca0d730b0da236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:51.703011 3498 raft_consensus.cc:1261] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 1 FOLLOWER]: Refusing update from remote peer 9a8f958b3bf642bd84ca0d730b0da236: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:51.703326 3699 consensus_queue.cc:1035] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f8528f7440a4e5b9659a996f49051c5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:51.703478 3696 consensus_queue.cc:1035] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [LEADER]: Connected to new peer: Peer: permanent_uuid: "399fb816f71e423e96dea2525a9950aa" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:51.705912 3703 mvcc.cc:205] Tried to move back new op lower bound from 7333814319932723200 to 7333814319899504640. Current Snapshot: MvccSnapshot[applied={T|T < 7333814319932723200}]
I20260927 04:06:51.706032 3702 mvcc.cc:205] Tried to move back new op lower bound from 7333814319932723200 to 7333814319899504640. Current Snapshot: MvccSnapshot[applied={T|T < 7333814319932723200}]
I20260927 04:06:51.708410 3224 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:51.708504 3224 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:51.709249 3224 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a792c651dfa4449839094cae388cd1f: Sending DeleteTablet for 3 replicas of tablet bc33ebcd6ca6450dba7ebcf35896eef3
I20260927 04:06:51.709559 3478 tablet_service.cc:1514] Processing DeleteTablet for tablet bc33ebcd6ca6450dba7ebcf35896eef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:51 UTC) from {username='slave'} at 127.0.0.1:54402
I20260927 04:06:51.709705 3609 tablet_service.cc:1514] Processing DeleteTablet for tablet bc33ebcd6ca6450dba7ebcf35896eef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:51 UTC) from {username='slave'} at 127.0.0.1:45818
I20260927 04:06:51.709717 3347 tablet_service.cc:1514] Processing DeleteTablet for tablet bc33ebcd6ca6450dba7ebcf35896eef3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:51 UTC) from {username='slave'} at 127.0.0.1:60072
I20260927 04:06:51.709980 3712 tablet_replica.cc:330] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: stopping tablet replica
I20260927 04:06:51.710115 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 3285
I20260927 04:06:51.712780 3712 raft_consensus.cc:2229] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:06:51.712909 3712 raft_consensus.cc:2258] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:51.713071 3711 tablet_replica.cc:330] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: stopping tablet replica
I20260927 04:06:51.713179 3711 raft_consensus.cc:2229] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:51.713285 3711 raft_consensus.cc:2258] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:51.713750 3712 ts_tablet_manager.cc:1905] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:51.715431 3712 ts_tablet_manager.cc:1918] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:51.715508 3712 log.cc:1196] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/bc33ebcd6ca6450dba7ebcf35896eef3
I20260927 04:06:51.715826 3712 ts_tablet_manager.cc:1939] T bc33ebcd6ca6450dba7ebcf35896eef3 P 9a8f958b3bf642bd84ca0d730b0da236: Deleting consensus metadata
I20260927 04:06:51.716234 3208 catalog_manager.cc:4841] TS 9a8f958b3bf642bd84ca0d730b0da236 (127.17.184.195:35171): tablet bc33ebcd6ca6450dba7ebcf35896eef3 (table test-table [id=a0baab34c97c470ba11ee5d38788057c]) successfully deleted
W20260927 04:06:51.718266 3208 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:33469 (error 108)
I20260927 04:06:51.718604 3711 ts_tablet_manager.cc:1905] T bc33ebcd6ca6450dba7ebcf35896eef3 P 3f8528f7440a4e5b9659a996f49051c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:51.718853 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.718722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33469 (local address 127.0.0.1:60076)
0927 04:06:51.718777 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.718821 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:06:51.719007 3208 catalog_manager.cc:4584] TS 399fb816f71e423e96dea2525a9950aa (127.17.184.193:33469): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc33ebcd6ca6450dba7ebcf35896eef3: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33469: connect: Connection refused (error 111)
I20260927 04:06:51.720160 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 3416
I20260927 04:06:51.726344 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 3547
I20260927 04:06:51.726408 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:51.726288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:45413 (local address 127.0.0.1:54404)
0927 04:06:51.726336 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:51.726370 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.194:45413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 04:06:51.731630 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 3193
2026-09-27T04:06:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (750 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:06:51 dist-test-slave-zqrn krb5kdc[3717](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:06:51 dist-test-slave-zqrn krb5kdc[3717](info): set up 2 sockets
Sep 27 04:06:51 dist-test-slave-zqrn krb5kdc[3717](info): commencing operation
krb5kdc: starting...
W20260927 04:06:52.774698 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:06:52 dist-test-slave-zqrn krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:06:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:06:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:06:52.898797 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:45977
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:45977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:52.981268 3730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:52.981521 3730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:52.981561 3730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:52.982743 3730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:52.982813 3730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:52.982836 3730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:52.982859 3730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:52.982887 3730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:52.984463 3730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:45977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:45977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:52.984757 3730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:52.984997 3730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:52.989971 3736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:52.989979 3738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:52.989979 3735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:52.990283 3730 server_base.cc:1034] running on GCE node
I20260927 04:06:52.990564 3730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:52.990942 3730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:52.992111 3730 hybrid_clock.cc:648] HybridClock initialized: now 1790482012992095 us; error 45 us; skew 500 ppm
Sep 27 04:06:52 dist-test-slave-zqrn krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482012, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:52.994632 3730 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:06:52.995144 3730 webserver.cc:460] Webserver started at http://127.17.184.254:43685/ using document root <none> and password file <none>
I20260927 04:06:52.995376 3730 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:52.995445 3730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:52.995591 3730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:52.996657 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "1858f2b45fd042a1be918f405eea30d1"
format_stamp: "Formatted at 2026-09-27 04:06:52 on dist-test-slave-zqrn"
server_key: "a61c15bac7f40bc64f8c4305b969ed62"
server_key_iv: "1322c589d7c0b0feebb1d431202fe216"
server_key_version: "encryptionkey@0"
I20260927 04:06:52.997008 3730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "1858f2b45fd042a1be918f405eea30d1"
format_stamp: "Formatted at 2026-09-27 04:06:52 on dist-test-slave-zqrn"
server_key: "a61c15bac7f40bc64f8c4305b969ed62"
server_key_iv: "1322c589d7c0b0feebb1d431202fe216"
server_key_version: "encryptionkey@0"
I20260927 04:06:52.998324 3730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:52.999069 3745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:52.999244 3730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:06:52.999341 3730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "1858f2b45fd042a1be918f405eea30d1"
format_stamp: "Formatted at 2026-09-27 04:06:52 on dist-test-slave-zqrn"
server_key: "a61c15bac7f40bc64f8c4305b969ed62"
server_key_iv: "1322c589d7c0b0feebb1d431202fe216"
server_key_version: "encryptionkey@0"
I20260927 04:06:52.999424 3730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:53.031283 3730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:53.033381 3730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:53.033560 3730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:53.037734 3730 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:45977
I20260927 04:06:53.037767 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:45977 every 8 connection(s)
I20260927 04:06:53.038120 3730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:53.039093 3798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:53.041517 3798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1: Bootstrap starting.
I20260927 04:06:53.042099 3798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:53.042392 3798 log.cc:824] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:53.043198 3798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1: No bootstrap required, opened a new log
I20260927 04:06:53.044070 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 3730
I20260927 04:06:53.044202 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:06:53.044423 18147 external_mini_cluster.cc:1428] Setting key 8c363f90edde21ec65a6692f9343c748
I20260927 04:06:53.045145 3798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858f2b45fd042a1be918f405eea30d1" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45977 } }
I20260927 04:06:53.045351 3798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:53.045387 3798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1858f2b45fd042a1be918f405eea30d1, State: Initialized, Role: FOLLOWER
I20260927 04:06:53.045500 3798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858f2b45fd042a1be918f405eea30d1" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45977 } }
I20260927 04:06:53.045579 3798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:53.045616 3798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:53.045655 3798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:53.046569 3798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858f2b45fd042a1be918f405eea30d1" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45977 } }
I20260927 04:06:53.046679 3798 leader_election.cc:302] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1858f2b45fd042a1be918f405eea30d1; no voters:
I20260927 04:06:53.046818 3798 leader_election.cc:288] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:06:53.046895 3803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:53.047062 3803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [term 1 LEADER]: Becoming Leader. State: Replica: 1858f2b45fd042a1be918f405eea30d1, State: Running, Role: LEADER
I20260927 04:06:53.047080 3798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:53.047155 3803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858f2b45fd042a1be918f405eea30d1" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45977 } }
I20260927 04:06:53.047451 3803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1858f2b45fd042a1be918f405eea30d1. Latest consensus state: current_term: 1 leader_uuid: "1858f2b45fd042a1be918f405eea30d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858f2b45fd042a1be918f405eea30d1" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45977 } } }
I20260927 04:06:53.047600 3803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:53.047462 3805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1858f2b45fd042a1be918f405eea30d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1858f2b45fd042a1be918f405eea30d1" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 45977 } } }
I20260927 04:06:53.047731 3805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1 [sys.catalog]: This master's current role is: LEADER
W20260927 04:06:53.048913 3818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:06:53.048976 3818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:06:53.049028 3812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:53.049170 3812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:06:53.050872 3812 catalog_manager.cc:1347] Generated new cluster ID: 7ef62189fe364c40a8c8164cb4ad0031
I20260927 04:06:53.050952 3812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:06:53 dist-test-slave-zqrn krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:53.056381 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.045037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.17.184.254:45977)
0927 04:06:53.045273 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.045282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.045315 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:06:53.045425 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.045432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.045486 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.045634 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:53.046451 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.047905 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.049010 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.049580 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.049701 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.052634 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:53.052639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:53.052639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:53.052651 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:53.054664 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.055079 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.055083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.055084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.055213 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.055416 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.055417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.055418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.055762 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:53.055947 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.056030 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.056179 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260927 04:06:53.097499 3812 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:06:53.098307 3812 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:53.103396 3812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1: Generated new TSK 0
I20260927 04:06:53.103686 3812 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:06:53.142283 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:45977
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:53.248199 3824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:53.248435 3824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:53.248473 3824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:53.249675 3824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:53.249744 3824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:53.249789 3824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:06:53.251572 3824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:53.251838 3824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:53.252082 3824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:53.257298 3830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:53.257306 3829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:53.257340 3824 server_base.cc:1034] running on GCE node
W20260927 04:06:53.257349 3832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:53.257792 3824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:53.258082 3824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:53.259267 3824 hybrid_clock.cc:648] HybridClock initialized: now 1790482013259256 us; error 39 us; skew 500 ppm
Sep 27 04:06:53 dist-test-slave-zqrn krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:53.261868 3824 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:06:53.262336 3824 webserver.cc:460] Webserver started at http://127.17.184.193:40413/ using document root <none> and password file <none>
I20260927 04:06:53.262583 3824 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:53.262655 3824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:53.262801 3824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:53.263892 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "6b1aff9c139d47eca3322f9e7db878ea"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "4f16116c4aed23392c67d6bf73b5b6a7"
server_key_iv: "da0dad0cbceca81ca4540ea37e0e193a"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.264253 3824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "6b1aff9c139d47eca3322f9e7db878ea"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "4f16116c4aed23392c67d6bf73b5b6a7"
server_key_iv: "da0dad0cbceca81ca4540ea37e0e193a"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.265671 3824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:06:53.266542 3839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.266781 3824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:06:53.266884 3824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "6b1aff9c139d47eca3322f9e7db878ea"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "4f16116c4aed23392c67d6bf73b5b6a7"
server_key_iv: "da0dad0cbceca81ca4540ea37e0e193a"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.266987 3824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:53.281718 3824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:53.283825 3824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:53.283993 3824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:53.284211 3824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:53.284525 3824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:53.284578 3824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.284621 3824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:53.284648 3824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.290417 3824 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:33237
I20260927 04:06:53.290495 3952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:33237 every 8 connection(s)
I20260927 04:06:53.290817 3824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
Sep 27 04:06:53 dist-test-slave-zqrn krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:53.298182 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 3824
I20260927 04:06:53.298305 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:06:53.298637 18147 external_mini_cluster.cc:1428] Setting key 653c3b4660c70913064dfc95599f9c8d
I20260927 04:06:53.300920 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.291574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:59341 (local address 127.17.184.254:45977)
0927 04:06:53.291618 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.291621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.291809 (+ 188us) server_negotiation.cc:407] Connection header received
0927 04:06:53.292310 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.292311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.292327 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.292385 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:53.293275 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.293793 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.294733 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.294982 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.295052 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.297347 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:53.297348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:53.297349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:53.297357 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:53.299256 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.299823 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.299825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.299826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.299890 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.300152 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.300153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.300154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.300308 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:53.300392 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.300688 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.300787 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:06:53.301429 3955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.291554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:45977 (local address 127.17.184.193:59341)
0927 04:06:53.291702 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.291738 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:53.292217 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:53.292440 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:53.292443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:53.292994 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:53.293185 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.293189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.293877 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.293878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.294656 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.294657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.295061 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.295062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.295160 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.295609 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:53.295617 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:53.296996 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:53.299434 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:53.299444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:53.299446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:53.299741 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:53.299965 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:53.299966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:53.299966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:53.300083 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:53.300381 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:53.300385 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:53.300515 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 04:06:53.300654 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 04:06:53.300854 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260927 04:06:53.302022 3953 heartbeater.cc:347] Connected to a master server at 127.17.184.254:45977
I20260927 04:06:53.302186 3953 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:53.302394 3953 heartbeater.cc:511] Master 127.17.184.254:45977 requested a full tablet report, sending...
I20260927 04:06:53.303100 3762 ts_manager.cc:194] Registered new tserver with Master: 6b1aff9c139d47eca3322f9e7db878ea (127.17.184.193:33237)
I20260927 04:06:53.303797 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:59341
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:06:53.327700 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:45977
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:53.410244 3958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:53.410493 3958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:53.410531 3958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:53.411723 3958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:53.411795 3958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:53.411840 3958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:06:53.413316 3958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:53.413616 3958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:53.413865 3958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:53.418987 3964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:53.418999 3963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:53.418995 3966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:53.419345 3958 server_base.cc:1034] running on GCE node
I20260927 04:06:53.419589 3958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:53.419843 3958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:53.421037 3958 hybrid_clock.cc:648] HybridClock initialized: now 1790482013421018 us; error 40 us; skew 500 ppm
Sep 27 04:06:53 dist-test-slave-zqrn krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:53.423856 3958 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:06:53.424324 3958 webserver.cc:460] Webserver started at http://127.17.184.194:35129/ using document root <none> and password file <none>
I20260927 04:06:53.424580 3958 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:53.424650 3958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:53.424798 3958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:53.425879 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "6335164dcd8b41188c9010882e0de8e5"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "e728fcca8eaf88ad1b7d1989fedecd05"
server_key_iv: "12e382fac6653291090152a5a1d5bf83"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.426246 3958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "6335164dcd8b41188c9010882e0de8e5"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "e728fcca8eaf88ad1b7d1989fedecd05"
server_key_iv: "12e382fac6653291090152a5a1d5bf83"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.427721 3958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:06:53.428580 3973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.428788 3958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:06:53.428905 3958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "6335164dcd8b41188c9010882e0de8e5"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "e728fcca8eaf88ad1b7d1989fedecd05"
server_key_iv: "12e382fac6653291090152a5a1d5bf83"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.428990 3958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:53.454325 3958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:53.456555 3958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:53.456737 3958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:53.456990 3958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:53.457305 3958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:53.457360 3958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.457402 3958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:53.457432 3958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.463696 3958 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:36787
I20260927 04:06:53.463742 4086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:36787 every 8 connection(s)
I20260927 04:06:53.464054 3958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:06:53 dist-test-slave-zqrn krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:53.473140 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 3958
I20260927 04:06:53.473246 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:06:53.473469 18147 external_mini_cluster.cc:1428] Setting key cd02d6e0a485a287315733a3d4f4e72f
I20260927 04:06:53.473789 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.464810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:55037 (local address 127.17.184.254:45977)
0927 04:06:53.464901 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.464903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.465095 (+ 192us) server_negotiation.cc:407] Connection header received
0927 04:06:53.465643 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.465644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.465658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.465698 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:53.466667 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.467235 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.468248 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.468471 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.468560 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.470788 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:53.470790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:53.470790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:53.470823 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:53.472213 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.472636 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.472637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.472638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.472700 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.472942 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.472943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.472943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.473096 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:53.473181 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.473595 (+ 414us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.473679 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:06:53.474025 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.464792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:45977 (local address 127.17.184.194:55037)
0927 04:06:53.464965 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.465009 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:53.465524 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:53.465791 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:53.465795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:53.466361 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:53.466555 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.466559 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.467329 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.467330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.468137 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.468140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.468570 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.468572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.468706 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.469197 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:53.469206 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:53.470617 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:53.472315 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:53.472327 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:53.472329 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:53.472546 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:53.472769 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:53.472770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:53.472770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:53.472876 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:53.473184 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:53.473191 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:53.473400 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 04:06:53.473564 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 04:06:53.473776 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260927 04:06:53.474493 4087 heartbeater.cc:347] Connected to a master server at 127.17.184.254:45977
I20260927 04:06:53.474576 4087 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:53.474771 4087 heartbeater.cc:511] Master 127.17.184.254:45977 requested a full tablet report, sending...
I20260927 04:06:53.475131 3762 ts_manager.cc:194] Registered new tserver with Master: 6335164dcd8b41188c9010882e0de8e5 (127.17.184.194:36787)
I20260927 04:06:53.475641 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:55037
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:06:53.498874 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:45977
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:53.579741 4092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:53.579964 4092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:53.579989 4092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:53.581182 4092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:53.581223 4092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:53.581250 4092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:06:53.582736 4092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:45977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:53.583014 4092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:53.583307 4092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:53.588433 4100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:53.588456 4098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:53.588470 4092 server_base.cc:1034] running on GCE node
W20260927 04:06:53.588479 4097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:53.588871 4092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:53.589135 4092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:53.590329 4092 hybrid_clock.cc:648] HybridClock initialized: now 1790482013590308 us; error 36 us; skew 500 ppm
Sep 27 04:06:53 dist-test-slave-zqrn krb5kdc[3717](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:53.593055 4092 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:06:53.593513 4092 webserver.cc:460] Webserver started at http://127.17.184.195:33741/ using document root <none> and password file <none>
I20260927 04:06:53.593758 4092 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:53.593806 4092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:53.593905 4092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:53.595070 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "fa69e7c62c964e66910605f03337b32a"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "014ec238527368585f04a0764396a283"
server_key_iv: "70a6e16dfc96cdd1c6f518a11ecfde93"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.595393 4092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "fa69e7c62c964e66910605f03337b32a"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "014ec238527368585f04a0764396a283"
server_key_iv: "70a6e16dfc96cdd1c6f518a11ecfde93"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.596735 4092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:06:53.597452 4107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.597620 4092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.597707 4092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "fa69e7c62c964e66910605f03337b32a"
format_stamp: "Formatted at 2026-09-27 04:06:53 on dist-test-slave-zqrn"
server_key: "014ec238527368585f04a0764396a283"
server_key_iv: "70a6e16dfc96cdd1c6f518a11ecfde93"
server_key_version: "encryptionkey@0"
I20260927 04:06:53.597760 4092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:53.609081 4092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:53.611272 4092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:53.611382 4092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:53.611562 4092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:53.611883 4092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:53.611917 4092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.611939 4092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:53.611989 4092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:53.617616 4092 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:43345
I20260927 04:06:53.617765 4220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:43345 every 8 connection(s)
I20260927 04:06:53.618016 4092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:06:53.624138 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 4092
I20260927 04:06:53.624250 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
Sep 27 04:06:53 dist-test-slave-zqrn krb5kdc[3717](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482013, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:53.624435 18147 external_mini_cluster.cc:1428] Setting key 2b64e81278594272752e8a5c69bc88a9
I20260927 04:06:53.627820 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.618824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:51211 (local address 127.17.184.254:45977)
0927 04:06:53.618921 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.618923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.619113 (+ 190us) server_negotiation.cc:407] Connection header received
0927 04:06:53.619608 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.619609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.619619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.619669 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:53.620666 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.621150 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.622235 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.622418 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.622474 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.624890 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:53.624892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:53.624892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:53.624900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:53.626383 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.626806 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.626808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.626809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.626867 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.627170 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.627171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.627172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.627294 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:53.627371 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.627640 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.627718 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:06:53.628059 4223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.618769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:45977 (local address 127.17.184.195:51211)
0927 04:06:53.618951 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.619017 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:53.619524 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:53.619733 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:53.619737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:53.620371 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:53.620583 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.620588 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.621242 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.621243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.622150 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.622152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.622499 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.622500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.622622 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.623178 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:53.623186 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:53.624789 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:53.626485 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:53.626492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:53.626495 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:53.626729 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:53.626979 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:53.626980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:53.626981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:53.627094 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:53.627362 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:53.627369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:53.627510 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 04:06:53.627618 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 04:06:53.627835 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260927 04:06:53.628580 4221 heartbeater.cc:347] Connected to a master server at 127.17.184.254:45977
I20260927 04:06:53.628705 4221 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:53.628919 4221 heartbeater.cc:511] Master 127.17.184.254:45977 requested a full tablet report, sending...
I20260927 04:06:53.629375 3762 ts_manager.cc:194] Registered new tserver with Master: fa69e7c62c964e66910605f03337b32a (127.17.184.195:43345)
I20260927 04:06:53.629738 3762 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:51211
I20260927 04:06:53.636399 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:06:53.644102 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.637760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.17.184.254:45977)
0927 04:06:53.637829 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.637831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.637970 (+ 139us) server_negotiation.cc:407] Connection header received
0927 04:06:53.638009 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.638010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.638019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.638070 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:53.638620 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.639224 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.639916 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.640118 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.640178 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.641528 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:53.641529 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:53.641530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:53.641537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:53.642887 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.643308 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.643309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.643309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.643362 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:53.643560 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:53.643560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:53.643561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:53.643726 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:53.643814 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.643912 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.643962 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:06:53.645723 3762 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:06:53.647713 3762 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:06:53.655025 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.650836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.17.184.194:36787)
0927 04:06:53.651048 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.651053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.651065 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:06:53.651105 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.651107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.651128 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.651249 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:53.651590 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.652322 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.653690 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.654506 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.654600 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.654665 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.654843 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.654915 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":113,"threads_started":1}
I20260927 04:06:53.655021 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.650369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:43345 (local address 127.0.0.1:43680)
0927 04:06:53.650625 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.650669 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:53.650751 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:53.651238 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:53.651241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:53.651243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:53.651324 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.651326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.652870 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.652871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.653940 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.653943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.654734 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.654735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.654800 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.654804 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:53.654834 (+ 30us) client_negotiation.cc:240] Negotiation successful
0927 04:06:53.654929 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":27,"thread_start_us":53,"threads_started":1}
I20260927 04:06:53.655014 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.650585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:36787 (local address 127.0.0.1:59354)
0927 04:06:53.650837 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.650849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:53.650930 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:53.651467 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:53.651468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:53.651470 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:53.651511 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.651513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.652440 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.652441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.653595 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.653597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.654610 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.654611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.654698 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.654714 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:06:53.654824 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 04:06:53.654922 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":108,"threads_started":1}
I20260927 04:06:53.655290 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.650490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.17.184.195:43345)
0927 04:06:53.651062 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.651067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.651077 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:06:53.651118 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.651120 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.651137 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.651236 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:53.651418 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.652759 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.654029 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.654624 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.655044 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.655103 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.655134 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.655187 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":126,"threads_started":1}
I20260927 04:06:53.655905 4021 tablet_service.cc:1467] Processing CreateTablet for tablet 558afd119ad144cca1c5fb20f02b0341 (DEFAULT_TABLE table=test-table [id=8b2956e3c40b40fbaf1ecef22fc35bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:53.656020 4155 tablet_service.cc:1467] Processing CreateTablet for tablet 558afd119ad144cca1c5fb20f02b0341 (DEFAULT_TABLE table=test-table [id=8b2956e3c40b40fbaf1ecef22fc35bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:53.656224 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.650369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33237 (local address 127.0.0.1:38434)
0927 04:06:53.650704 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.650714 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:53.650760 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:53.651585 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:53.651585 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:53.651586 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:53.651616 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.651617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.652993 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.652994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.654263 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.654266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.655929 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.655930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.656001 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.656005 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:53.656064 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 04:06:53.656100 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":3,"thread_start_us":88,"threads_started":1}
I20260927 04:06:53.656286 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558afd119ad144cca1c5fb20f02b0341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:53.656319 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.650609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.17.184.193:33237)
0927 04:06:53.651370 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.651375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.651383 (+ 8us) server_negotiation.cc:407] Connection header received
0927 04:06:53.651422 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.651424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.651440 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.651511 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:53.651703 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.652879 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.654353 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.655773 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.655844 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.655894 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.656146 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.656199 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":84,"threads_started":1}
I20260927 04:06:53.656425 4155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558afd119ad144cca1c5fb20f02b0341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:53.657167 3887 tablet_service.cc:1467] Processing CreateTablet for tablet 558afd119ad144cca1c5fb20f02b0341 (DEFAULT_TABLE table=test-table [id=8b2956e3c40b40fbaf1ecef22fc35bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:53.657496 3887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 558afd119ad144cca1c5fb20f02b0341. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:53.658774 4238 tablet_bootstrap.cc:492] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Bootstrap starting.
I20260927 04:06:53.659359 4238 tablet_bootstrap.cc:654] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:53.659756 4238 log.cc:824] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:53.660014 4239 tablet_bootstrap.cc:492] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Bootstrap starting.
I20260927 04:06:53.660501 4239 tablet_bootstrap.cc:654] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:53.661017 4241 tablet_bootstrap.cc:492] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea: Bootstrap starting.
I20260927 04:06:53.661036 4239 log.cc:824] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:53.661725 4238 tablet_bootstrap.cc:492] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: No bootstrap required, opened a new log
I20260927 04:06:53.661779 4238 ts_tablet_manager.cc:1397] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 04:06:53.661754 4239 tablet_bootstrap.cc:492] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: No bootstrap required, opened a new log
I20260927 04:06:53.661844 4239 ts_tablet_manager.cc:1397] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:06:53.663164 4241 tablet_bootstrap.cc:654] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:53.663394 4239 raft_consensus.cc:348] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.663399 4238 raft_consensus.cc:348] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.663569 4239 raft_consensus.cc:374] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:53.663589 4238 raft_consensus.cc:374] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:53.663626 4239 raft_consensus.cc:729] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa69e7c62c964e66910605f03337b32a, State: Initialized, Role: FOLLOWER
I20260927 04:06:53.663627 4238 raft_consensus.cc:729] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6335164dcd8b41188c9010882e0de8e5, State: Initialized, Role: FOLLOWER
I20260927 04:06:53.663704 4241 log.cc:824] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:53.663767 4238 consensus_queue.cc:260] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.663766 4239 consensus_queue.cc:260] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.664044 4238 ts_tablet_manager.cc:1428] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:53.664070 4239 ts_tablet_manager.cc:1428] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:53.664137 4221 heartbeater.cc:503] Master 127.17.184.254:45977 was elected leader, sending a full tablet report...
I20260927 04:06:53.664153 4087 heartbeater.cc:503] Master 127.17.184.254:45977 was elected leader, sending a full tablet report...
I20260927 04:06:53.664948 4241 tablet_bootstrap.cc:492] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea: No bootstrap required, opened a new log
I20260927 04:06:53.665014 4241 ts_tablet_manager.cc:1397] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 04:06:53.666373 4241 raft_consensus.cc:348] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.666502 4241 raft_consensus.cc:374] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:53.666523 4241 raft_consensus.cc:729] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b1aff9c139d47eca3322f9e7db878ea, State: Initialized, Role: FOLLOWER
I20260927 04:06:53.666633 4241 consensus_queue.cc:260] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.666932 4241 ts_tablet_manager.cc:1428] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:53.666970 3953 heartbeater.cc:503] Master 127.17.184.254:45977 was elected leader, sending a full tablet report...
W20260927 04:06:53.714679 4088 tablet.cc:2369] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:06:53.791735 3954 tablet.cc:2369] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:06:53.832546 4244 raft_consensus.cc:482] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:53.832679 4244 raft_consensus.cc:504] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.833107 4244 leader_election.cc:288] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b1aff9c139d47eca3322f9e7db878ea (127.17.184.193:33237), fa69e7c62c964e66910605f03337b32a (127.17.184.195:43345)
I20260927 04:06:53.836177 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.833392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:37751 (local address 127.17.184.193:33237)
0927 04:06:53.833455 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.833458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.833634 (+ 176us) server_negotiation.cc:407] Connection header received
0927 04:06:53.833793 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.833794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.833807 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.833882 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:53.834062 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.834543 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.835470 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.835817 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.835910 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.835919 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.836068 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.836100 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 04:06:53.836186 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.833323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33237 (local address 127.17.184.194:37751)
0927 04:06:53.833573 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.833584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:53.833684 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:53.833887 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:53.833888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:53.833890 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:53.833965 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.833968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.834628 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.834629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.835381 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.835383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.835912 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.835912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.835989 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.835992 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:53.836061 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 04:06:53.836102 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":94,"thread_start_us":97,"threads_started":1}
I20260927 04:06:53.836225 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.833470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:52345 (local address 127.17.184.195:43345)
0927 04:06:53.833524 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.833526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.833537 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:06:53.833621 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.833623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.833633 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.833687 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:53.833901 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.834463 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.835489 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.835858 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.835945 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.835953 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.836110 (+ 157us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.836148 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 04:06:53.836261 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.833323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:43345 (local address 127.17.184.194:52345)
0927 04:06:53.833382 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.833398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:53.833468 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:53.833703 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:53.833704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:53.833706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:53.833794 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.833796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.834556 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.834557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.835372 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:53.835374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.835942 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.835943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:53.836029 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.836032 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:53.836104 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 04:06:53.836159 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 04:06:53.836625 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558afd119ad144cca1c5fb20f02b0341" candidate_uuid: "6335164dcd8b41188c9010882e0de8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa69e7c62c964e66910605f03337b32a" is_pre_election: true
I20260927 04:06:53.836737 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558afd119ad144cca1c5fb20f02b0341" candidate_uuid: "6335164dcd8b41188c9010882e0de8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1aff9c139d47eca3322f9e7db878ea" is_pre_election: true
I20260927 04:06:53.836814 4175 raft_consensus.cc:2454] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6335164dcd8b41188c9010882e0de8e5 in term 0.
I20260927 04:06:53.836897 3907 raft_consensus.cc:2454] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6335164dcd8b41188c9010882e0de8e5 in term 0.
I20260927 04:06:53.837036 3977 leader_election.cc:302] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6335164dcd8b41188c9010882e0de8e5, fa69e7c62c964e66910605f03337b32a; no voters:
I20260927 04:06:53.837213 4244 raft_consensus.cc:2781] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:06:53.837268 4244 raft_consensus.cc:482] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:06:53.837296 4244 raft_consensus.cc:3037] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:53.837990 4244 raft_consensus.cc:504] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.838228 4244 leader_election.cc:288] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [CANDIDATE]: Term 1 election: Requested vote from peers 6b1aff9c139d47eca3322f9e7db878ea (127.17.184.193:33237), fa69e7c62c964e66910605f03337b32a (127.17.184.195:43345)
I20260927 04:06:53.838291 3907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558afd119ad144cca1c5fb20f02b0341" candidate_uuid: "6335164dcd8b41188c9010882e0de8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b1aff9c139d47eca3322f9e7db878ea"
I20260927 04:06:53.838299 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "558afd119ad144cca1c5fb20f02b0341" candidate_uuid: "6335164dcd8b41188c9010882e0de8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa69e7c62c964e66910605f03337b32a"
I20260927 04:06:53.838376 4175 raft_consensus.cc:3037] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:53.838402 3907 raft_consensus.cc:3037] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:53.839151 4175 raft_consensus.cc:2454] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6335164dcd8b41188c9010882e0de8e5 in term 1.
I20260927 04:06:53.839308 3907 raft_consensus.cc:2454] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6335164dcd8b41188c9010882e0de8e5 in term 1.
I20260927 04:06:53.839377 3977 leader_election.cc:302] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6335164dcd8b41188c9010882e0de8e5, fa69e7c62c964e66910605f03337b32a; no voters:
I20260927 04:06:53.839545 4244 raft_consensus.cc:2781] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:53.839737 4244 raft_consensus.cc:686] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 1 LEADER]: Becoming Leader. State: Replica: 6335164dcd8b41188c9010882e0de8e5, State: Running, Role: LEADER
I20260927 04:06:53.839840 4244 consensus_queue.cc:237] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } }
I20260927 04:06:53.840688 3762 catalog_manager.cc:5495] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6335164dcd8b41188c9010882e0de8e5 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "6335164dcd8b41188c9010882e0de8e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6335164dcd8b41188c9010882e0de8e5" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 36787 } health_report { overall_health: HEALTHY } } }
W20260927 04:06:53.868614 4222 tablet.cc:2369] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:06:53.883690 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.881398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.17.184.194:36787)
0927 04:06:53.881458 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:53.881465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:53.881513 (+ 48us) server_negotiation.cc:407] Connection header received
0927 04:06:53.881548 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:53.881548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:53.881565 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:53.881626 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:06:53.881770 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.882250 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.883007 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:53.883195 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:53.883251 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:53.883369 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:06:53.883453 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:06:53.883571 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:53.883604 (+ 33us) server_negotiation.cc:299] Negotiation successful
0927 04:06:53.883614 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:06:53.885509 4175 raft_consensus.cc:1261] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 1 FOLLOWER]: Refusing update from remote peer 6335164dcd8b41188c9010882e0de8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:53.885547 3907 raft_consensus.cc:1261] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 1 FOLLOWER]: Refusing update from remote peer 6335164dcd8b41188c9010882e0de8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:53.885854 4244 consensus_queue.cc:1035] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa69e7c62c964e66910605f03337b32a" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 43345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:53.886030 4248 consensus_queue.cc:1035] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b1aff9c139d47eca3322f9e7db878ea" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:53.888921 4253 mvcc.cc:205] Tried to move back new op lower bound from 7333814328872079360 to 7333814328688570368. Current Snapshot: MvccSnapshot[applied={T|T < 7333814328872079360}]
I20260927 04:06:53.889271 4252 mvcc.cc:205] Tried to move back new op lower bound from 7333814328872079360 to 7333814328688570368. Current Snapshot: MvccSnapshot[applied={T|T < 7333814328872079360 or (T in {7333814328872079360})}]
I20260927 04:06:53.890730 3762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:53.890832 3762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:53.891745 3762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1858f2b45fd042a1be918f405eea30d1: Sending DeleteTablet for 3 replicas of tablet 558afd119ad144cca1c5fb20f02b0341
I20260927 04:06:53.892081 3887 tablet_service.cc:1514] Processing DeleteTablet for tablet 558afd119ad144cca1c5fb20f02b0341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:53 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:38434
I20260927 04:06:53.892108 4155 tablet_service.cc:1514] Processing DeleteTablet for tablet 558afd119ad144cca1c5fb20f02b0341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:53 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:43680
I20260927 04:06:53.892390 4260 tablet_replica.cc:330] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: stopping tablet replica
I20260927 04:06:53.892365 4021 tablet_service.cc:1514] Processing DeleteTablet for tablet 558afd119ad144cca1c5fb20f02b0341 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:53 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:59354
I20260927 04:06:53.892568 4261 tablet_replica.cc:330] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea: stopping tablet replica
I20260927 04:06:53.892611 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 3824
I20260927 04:06:53.892705 4260 raft_consensus.cc:2229] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:53.892832 4261 raft_consensus.cc:2229] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:53.892997 4261 raft_consensus.cc:2258] T 558afd119ad144cca1c5fb20f02b0341 P 6b1aff9c139d47eca3322f9e7db878ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:53.893229 4260 raft_consensus.cc:2258] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:53.893832 4262 tablet_replica.cc:330] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: stopping tablet replica
I20260927 04:06:53.893975 4262 raft_consensus.cc:2229] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:06:53.894109 4262 raft_consensus.cc:2258] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:53.894603 4262 ts_tablet_manager.cc:1905] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:53.894640 4260 ts_tablet_manager.cc:1905] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:53.896371 4260 ts_tablet_manager.cc:1918] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:53.896437 4260 log.cc:1196] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/558afd119ad144cca1c5fb20f02b0341
I20260927 04:06:53.896713 4260 ts_tablet_manager.cc:1939] T 558afd119ad144cca1c5fb20f02b0341 P fa69e7c62c964e66910605f03337b32a: Deleting consensus metadata
I20260927 04:06:53.896709 4262 ts_tablet_manager.cc:1918] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:53.896772 4262 log.cc:1196] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/558afd119ad144cca1c5fb20f02b0341
I20260927 04:06:53.897027 4262 ts_tablet_manager.cc:1939] T 558afd119ad144cca1c5fb20f02b0341 P 6335164dcd8b41188c9010882e0de8e5: Deleting consensus metadata
I20260927 04:06:53.897114 3749 catalog_manager.cc:4841] TS fa69e7c62c964e66910605f03337b32a (127.17.184.195:43345): tablet 558afd119ad144cca1c5fb20f02b0341 (table test-table [id=8b2956e3c40b40fbaf1ecef22fc35bde]) successfully deleted
I20260927 04:06:53.897372 3747 catalog_manager.cc:4841] TS 6335164dcd8b41188c9010882e0de8e5 (127.17.184.194:36787): tablet 558afd119ad144cca1c5fb20f02b0341 (table test-table [id=8b2956e3c40b40fbaf1ecef22fc35bde]) successfully deleted
W20260927 04:06:53.899282 3747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:33237 (error 108)
I20260927 04:06:53.899475 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 3958
I20260927 04:06:53.900516 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:53.900373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:33237 (local address 127.0.0.1:38442)
0927 04:06:53.900431 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:53.900479 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:06:53.900746 3747 catalog_manager.cc:4584] TS 6b1aff9c139d47eca3322f9e7db878ea (127.17.184.193:33237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 558afd119ad144cca1c5fb20f02b0341: Network error: Client connection negotiation failed: client connection to 127.17.184.193:33237: connect: Connection refused (error 111)
I20260927 04:06:53.904263 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 4092
I20260927 04:06:53.909099 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 3730
2026-09-27T04:06:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2177 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T04:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:06:53Z Disabled control of system clock
I20260927 04:06:53.936582 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44979
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:54.019456 4267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:54.019686 4267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:54.019707 4267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:54.021080 4267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:54.021130 4267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:54.021143 4267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:54.021157 4267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:54.021168 4267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:54.022682 4267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:54.023016 4267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:54.023347 4267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:06:54.028851 4267 server_base.cc:1034] running on GCE node
W20260927 04:06:54.028782 4272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:54.028761 4275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:54.028765 4273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:54.029361 4267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:54.029692 4267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:54.030948 4267 hybrid_clock.cc:648] HybridClock initialized: now 1790482014030882 us; error 46 us; skew 500 ppm
I20260927 04:06:54.038354 4267 webserver.cc:460] Webserver started at http://127.17.184.254:34365/ using document root <none> and password file <none>
I20260927 04:06:54.038671 4267 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:54.038753 4267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:54.038954 4267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:54.040098 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "abc294b0821c4538bcb9b981c017e98b"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "eafbf3410560a35f9dffc1115aaf4160"
server_key_iv: "3cc23c8449a04d270baa8ba326375be3"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.040519 4267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "abc294b0821c4538bcb9b981c017e98b"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "eafbf3410560a35f9dffc1115aaf4160"
server_key_iv: "3cc23c8449a04d270baa8ba326375be3"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.041956 4267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260927 04:06:54.043035 4281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.043264 4267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:54.043366 4267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "abc294b0821c4538bcb9b981c017e98b"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "eafbf3410560a35f9dffc1115aaf4160"
server_key_iv: "3cc23c8449a04d270baa8ba326375be3"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.043459 4267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:54.052294 4267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:54.052704 4267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:54.052856 4267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:54.056941 4267 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44979
I20260927 04:06:54.056983 4333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44979 every 8 connection(s)
I20260927 04:06:54.057337 4267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:54.058339 4334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:54.060812 4334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b: Bootstrap starting.
I20260927 04:06:54.060812 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 4267
I20260927 04:06:54.061000 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:06:54.061228 18147 external_mini_cluster.cc:1428] Setting key c0d1d96b2f4a8975b7d5eb3b70856b4a
I20260927 04:06:54.061688 4334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:54.062103 4334 log.cc:824] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:54.062884 4334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b: No bootstrap required, opened a new log
I20260927 04:06:54.065282 4334 raft_consensus.cc:348] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc294b0821c4538bcb9b981c017e98b" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44979 } }
I20260927 04:06:54.065445 4334 raft_consensus.cc:374] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:54.065482 4334 raft_consensus.cc:729] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abc294b0821c4538bcb9b981c017e98b, State: Initialized, Role: FOLLOWER
I20260927 04:06:54.065599 4334 consensus_queue.cc:260] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc294b0821c4538bcb9b981c017e98b" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44979 } }
I20260927 04:06:54.065670 4334 raft_consensus.cc:388] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:54.065701 4334 raft_consensus.cc:482] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:54.065745 4334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:54.066457 4334 raft_consensus.cc:504] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc294b0821c4538bcb9b981c017e98b" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44979 } }
I20260927 04:06:54.066547 4334 leader_election.cc:302] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abc294b0821c4538bcb9b981c017e98b; no voters:
I20260927 04:06:54.066783 4334 leader_election.cc:288] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:06:54.066819 4339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:54.066941 4339 raft_consensus.cc:686] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [term 1 LEADER]: Becoming Leader. State: Replica: abc294b0821c4538bcb9b981c017e98b, State: Running, Role: LEADER
I20260927 04:06:54.067035 4339 consensus_queue.cc:237] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc294b0821c4538bcb9b981c017e98b" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44979 } }
I20260927 04:06:54.067099 4334 sys_catalog.cc:564] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:54.067363 4339 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abc294b0821c4538bcb9b981c017e98b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc294b0821c4538bcb9b981c017e98b" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44979 } } }
I20260927 04:06:54.067513 4339 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:54.067708 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.061861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54684 (local address 127.17.184.254:44979)
0927 04:06:54.062102 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.062111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.062144 (+ 33us) server_negotiation.cc:407] Connection header received
0927 04:06:54.062762 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.062771 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.062859 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.063037 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:54.063256 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.064683 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.065451 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.066066 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.066189 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.066947 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:54.066951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:54.066951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:54.067164 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:54.067206 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:54.067220 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:54.067308 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.067375 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.067457 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260927 04:06:54.067749 4339 sys_catalog.cc:455] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [sys.catalog]: SysCatalogTable state changed. Reason: New leader abc294b0821c4538bcb9b981c017e98b. Latest consensus state: current_term: 1 leader_uuid: "abc294b0821c4538bcb9b981c017e98b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abc294b0821c4538bcb9b981c017e98b" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44979 } } }
I20260927 04:06:54.067803 4346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:54.067809 4339 sys_catalog.cc:458] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:54.068317 4346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:06:54.069802 4346 catalog_manager.cc:1347] Generated new cluster ID: c53deb3681394dc099506391f329efa6
I20260927 04:06:54.069864 4346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:06:54.075323 4346 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:06:54.075887 4346 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:54.078380 4346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b: Generated new TSK 0
I20260927 04:06:54.078558 4346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:06:54.086560 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44979
--builtin_ntp_servers=127.17.184.212:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:54.210985 4358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:54.211231 4358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:54.211266 4358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:54.212548 4358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:54.212610 4358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:54.212653 4358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:06:54.214031 4358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:54.214294 4358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:54.214597 4358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:54.220870 4364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:54.220870 4366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:54.220872 4363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:54.221179 4358 server_base.cc:1034] running on GCE node
I20260927 04:06:54.221405 4358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:54.221649 4358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:54.222816 4358 hybrid_clock.cc:648] HybridClock initialized: now 1790482014222804 us; error 30 us; skew 500 ppm
I20260927 04:06:54.224120 4358 webserver.cc:460] Webserver started at http://127.17.184.193:44989/ using document root <none> and password file <none>
I20260927 04:06:54.224381 4358 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:54.224450 4358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:54.224623 4358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:54.225725 4358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "4ba2010a96844bedb420a33b1bab7a13"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "78ad41363e51d0e48d679cfd6dbf20e7"
server_key_iv: "72ec5b1a1de590285f51a19a11d194c1"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.226076 4358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "4ba2010a96844bedb420a33b1bab7a13"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "78ad41363e51d0e48d679cfd6dbf20e7"
server_key_iv: "72ec5b1a1de590285f51a19a11d194c1"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.227511 4358 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:54.228327 4372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.228502 4358 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:54.228595 4358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "4ba2010a96844bedb420a33b1bab7a13"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "78ad41363e51d0e48d679cfd6dbf20e7"
server_key_iv: "72ec5b1a1de590285f51a19a11d194c1"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.228720 4358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:54.237227 4358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:54.237531 4358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:54.237686 4358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:54.237901 4358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:54.238216 4358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:54.238271 4358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.238313 4358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:54.238339 4358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.243966 4358 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:38133
I20260927 04:06:54.244033 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:38133 every 8 connection(s)
I20260927 04:06:54.244393 4358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:06:54.250437 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.245149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:39263 (local address 127.17.184.254:44979)
0927 04:06:54.245224 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.245227 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.245440 (+ 213us) server_negotiation.cc:407] Connection header received
0927 04:06:54.245978 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.245979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.245990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.246082 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:54.246923 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.247769 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.248721 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.249051 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.249134 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.250109 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:54.250111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:54.250112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:54.250119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:54.250138 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:54.250143 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:54.250213 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.250327 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.250352 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:06:54.250738 4488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.245113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44979 (local address 127.17.184.193:39263)
0927 04:06:54.245290 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.245347 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:54.245884 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:54.246139 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:54.246142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:54.246172 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:54.246793 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.246798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.247859 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.247859 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.248631 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.248633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.249169 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.249170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.249281 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.249944 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:54.249945 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:54.249974 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:54.249996 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:54.250233 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:54.250238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:54.250241 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:54.250332 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 04:06:54.250454 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260927 04:06:54.251314 4486 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44979
I20260927 04:06:54.251421 4486 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:54.251624 4486 heartbeater.cc:511] Master 127.17.184.254:44979 requested a full tablet report, sending...
I20260927 04:06:54.252097 4298 ts_manager.cc:194] Registered new tserver with Master: 4ba2010a96844bedb420a33b1bab7a13 (127.17.184.193:38133)
I20260927 04:06:54.252774 4298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.193:39263
I20260927 04:06:54.252898 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 4358
I20260927 04:06:54.253005 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:06:54.253237 18147 external_mini_cluster.cc:1428] Setting key 52876b1c147bfacea74db6d747950acd
I20260927 04:06:54.254065 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44979
--builtin_ntp_servers=127.17.184.212:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:54.335507 4489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:54.335713 4489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:54.335736 4489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:54.336885 4489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:54.336926 4489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:54.336951 4489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:06:54.338299 4489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:54.338546 4489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:54.338871 4489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:54.343983 4495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:54.343995 4494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:54.343999 4497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:54.344252 4489 server_base.cc:1034] running on GCE node
I20260927 04:06:54.344460 4489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:54.344749 4489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:54.345942 4489 hybrid_clock.cc:648] HybridClock initialized: now 1790482014345932 us; error 51 us; skew 500 ppm
I20260927 04:06:54.347301 4489 webserver.cc:460] Webserver started at http://127.17.184.194:35869/ using document root <none> and password file <none>
I20260927 04:06:54.347549 4489 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:54.347616 4489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:54.347761 4489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:54.348825 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "84058544041a4171bdfb4af79cfa6748"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "ac1e5afe56d412f7ccbaba1cb95c2608"
server_key_iv: "80317b7220ad151b471b44635d55c40c"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.349181 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "84058544041a4171bdfb4af79cfa6748"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "ac1e5afe56d412f7ccbaba1cb95c2608"
server_key_iv: "80317b7220ad151b471b44635d55c40c"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.350538 4489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:54.351380 4503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.351576 4489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 04:06:54.351678 4489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "84058544041a4171bdfb4af79cfa6748"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "ac1e5afe56d412f7ccbaba1cb95c2608"
server_key_iv: "80317b7220ad151b471b44635d55c40c"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.351766 4489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:54.388394 4489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:54.388743 4489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:54.388902 4489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:54.389153 4489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:54.389485 4489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:54.389540 4489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.389585 4489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:54.389609 4489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.395341 4489 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:43165
I20260927 04:06:54.395382 4616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:43165 every 8 connection(s)
I20260927 04:06:54.395697 4489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:06:54.399540 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 4489
I20260927 04:06:54.399626 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:06:54.399823 18147 external_mini_cluster.cc:1428] Setting key 863470d47cfe38dde690903693760c22
I20260927 04:06:54.400873 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44979
--builtin_ntp_servers=127.17.184.212:34699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 04:06:54.401247 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.396435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:58191 (local address 127.17.184.254:44979)
0927 04:06:54.396502 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.396504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.396752 (+ 248us) server_negotiation.cc:407] Connection header received
0927 04:06:54.397252 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.397253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.397262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.397317 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:54.398030 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.398523 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.399701 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.399940 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.400009 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.400950 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:54.400951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:54.400952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:54.400957 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:54.400974 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:54.400979 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:54.401038 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.401164 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.401187 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:06:54.401518 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.396429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44979 (local address 127.17.184.194:58191)
0927 04:06:54.396625 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.396659 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:54.397156 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:54.397387 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:54.397390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:54.397398 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:54.397953 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.397958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.398709 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.398710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.399606 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.399609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.400017 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.400018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.400154 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.400815 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:54.400816 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:54.400844 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:54.400865 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:54.401065 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:54.401071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:54.401075 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:54.401136 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 04:06:54.401253 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260927 04:06:54.402029 4617 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44979
I20260927 04:06:54.402136 4617 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:54.402344 4617 heartbeater.cc:511] Master 127.17.184.254:44979 requested a full tablet report, sending...
I20260927 04:06:54.402732 4298 ts_manager.cc:194] Registered new tserver with Master: 84058544041a4171bdfb4af79cfa6748 (127.17.184.194:43165)
I20260927 04:06:54.403146 4298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.194:58191
W20260927 04:06:54.483686 4620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:54.483940 4620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:54.483975 4620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:54.485106 4620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:54.485193 4620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:54.485244 4620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:06:54.486637 4620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:34699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:54.486923 4620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:54.487237 4620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:54.492379 4626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:54.492370 4628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:54.492406 4620 server_base.cc:1034] running on GCE node
W20260927 04:06:54.492369 4625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:54.492825 4620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:54.493044 4620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:54.494195 4620 hybrid_clock.cc:648] HybridClock initialized: now 1790482014494180 us; error 31 us; skew 500 ppm
I20260927 04:06:54.495571 4620 webserver.cc:460] Webserver started at http://127.17.184.195:33939/ using document root <none> and password file <none>
I20260927 04:06:54.495808 4620 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:54.495852 4620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:54.496012 4620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:54.497044 4620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "bcef3846848c4110afac2bba53ea60f4"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "094bb6e58d702f5c06b74c8cc619bf22"
server_key_iv: "e1f4a07804834bf2e811336ec7efff0a"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.497396 4620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "bcef3846848c4110afac2bba53ea60f4"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "094bb6e58d702f5c06b74c8cc619bf22"
server_key_iv: "e1f4a07804834bf2e811336ec7efff0a"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.498673 4620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:06:54.499405 4634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.499569 4620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.499663 4620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "bcef3846848c4110afac2bba53ea60f4"
format_stamp: "Formatted at 2026-09-27 04:06:54 on dist-test-slave-zqrn"
server_key: "094bb6e58d702f5c06b74c8cc619bf22"
server_key_iv: "e1f4a07804834bf2e811336ec7efff0a"
server_key_version: "encryptionkey@0"
I20260927 04:06:54.499745 4620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:54.518682 4620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:54.519034 4620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:54.519196 4620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:54.519430 4620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:54.519750 4620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:54.519804 4620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.519842 4620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:54.519870 4620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:54.525885 4620 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:46249
I20260927 04:06:54.525919 4747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:46249 every 8 connection(s)
I20260927 04:06:54.526266 4620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:06:54.531859 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.527087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:52813 (local address 127.17.184.254:44979)
0927 04:06:54.527133 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.527136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.527368 (+ 232us) server_negotiation.cc:407] Connection header received
0927 04:06:54.528015 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.528016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.528029 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.528081 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:54.528765 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.529344 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.530356 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.530569 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.530645 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.531527 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:54.531529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:54.531529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:54.531535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:54.531551 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:54.531555 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:54.531612 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.531750 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.531778 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:06:54.532145 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.527075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44979 (local address 127.17.184.195:52813)
0927 04:06:54.527248 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.527280 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:54.527860 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:54.528151 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:54.528155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:54.528165 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:54.528674 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.528679 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.529439 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.529440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.530263 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.530265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.530663 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.530664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.530827 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.531353 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:54.531354 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:54.531382 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:54.531405 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:54.531617 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:54.531621 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:54.531625 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:54.531751 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 04:06:54.531877 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260927 04:06:54.532625 4748 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44979
I20260927 04:06:54.532723 4748 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:54.532864 4748 heartbeater.cc:511] Master 127.17.184.254:44979 requested a full tablet report, sending...
I20260927 04:06:54.533226 4298 ts_manager.cc:194] Registered new tserver with Master: bcef3846848c4110afac2bba53ea60f4 (127.17.184.195:46249)
I20260927 04:06:54.533608 4298 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.195:52813
I20260927 04:06:54.536125 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 4620
I20260927 04:06:54.536237 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:06:54.536445 18147 external_mini_cluster.cc:1428] Setting key 23619ccfa75a05762c9d66a6ec339508
I20260927 04:06:54.537331 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:06:54.541594 4337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.538711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.17.184.254:44979)
0927 04:06:54.538752 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.538754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.538868 (+ 114us) server_negotiation.cc:407] Connection header received
0927 04:06:54.538933 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.538934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.538944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.539004 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:54.539146 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.539610 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.540366 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.540574 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.540659 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.541280 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:54.541281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:54.541282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:54.541286 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:54.541299 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:54.541303 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:54.541412 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.541503 (+ 91us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.541514 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:06:54.543083 4298 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:06:54.544788 4298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:06:54.551014 4758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.546835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:38133 (local address 127.0.0.1:46974)
0927 04:06:54.546998 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.547006 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:54.547066 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:54.547672 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:54.547673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:54.547675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:54.547718 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.547719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.549044 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.549045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.550066 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.550068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.550737 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.550738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.550798 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.550803 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:06:54.550859 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 04:06:54.550923 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260927 04:06:54.551008 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.547098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.17.184.193:38133)
0927 04:06:54.547508 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.547512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.547521 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:54.547556 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.547558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.547572 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.547789 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:54.547846 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.548955 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.550151 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.550633 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.550727 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.550813 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.550868 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.550924 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":93,"threads_started":1}
I20260927 04:06:54.551414 4757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.546831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:46249 (local address 127.0.0.1:60942)
0927 04:06:54.546959 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.547000 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:54.547066 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:54.547402 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:54.547405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:54.547407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:54.547480 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.547482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.548570 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.548571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.549672 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.549674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.550678 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.550679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.550755 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.550766 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 04:06:54.550828 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 04:06:54.550876 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260927 04:06:54.551577 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.547067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.17.184.195:46249)
0927 04:06:54.547239 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.547243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.547254 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:06:54.547290 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.547292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.547306 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.547400 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:54.547678 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.548483 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.549766 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.550484 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.550552 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.550634 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.550967 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.551007 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260927 04:06:54.551756 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 75ffe6e9bffd417d92a9fc3981db2fd9 (DEFAULT_TABLE table=test-table [id=8f8c1adae829486f8a37463567a12c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:54.552146 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ffe6e9bffd417d92a9fc3981db2fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:54.552182 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 75ffe6e9bffd417d92a9fc3981db2fd9 (DEFAULT_TABLE table=test-table [id=8f8c1adae829486f8a37463567a12c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:54.552624 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ffe6e9bffd417d92a9fc3981db2fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:54.552646 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.546831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:43165 (local address 127.0.0.1:52618)
0927 04:06:54.547168 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.547177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:54.547216 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:54.547740 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:54.547741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:54.547743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:54.547783 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.547784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.550156 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.550157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.551379 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.551381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.552408 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.552409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.552468 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.552473 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:06:54.552533 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 04:06:54.552580 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260927 04:06:54.552886 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.547285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.17.184.194:43165)
0927 04:06:54.547457 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.547463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.547474 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:06:54.547518 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.547520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.547556 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.547655 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:54.547840 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.548955 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.551464 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.552214 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.552290 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.552372 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.552562 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.552614 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":105,"threads_started":1}
I20260927 04:06:54.553670 4551 tablet_service.cc:1467] Processing CreateTablet for tablet 75ffe6e9bffd417d92a9fc3981db2fd9 (DEFAULT_TABLE table=test-table [id=8f8c1adae829486f8a37463567a12c6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:54.553985 4551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75ffe6e9bffd417d92a9fc3981db2fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:54.555382 4762 tablet_bootstrap.cc:492] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13: Bootstrap starting.
I20260927 04:06:54.555933 4763 tablet_bootstrap.cc:492] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Bootstrap starting.
I20260927 04:06:54.556116 4762 tablet_bootstrap.cc:654] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:54.556492 4762 log.cc:824] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:54.556612 4763 tablet_bootstrap.cc:654] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:54.556947 4763 log.cc:824] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:54.557286 4762 tablet_bootstrap.cc:492] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13: No bootstrap required, opened a new log
I20260927 04:06:54.557364 4762 ts_tablet_manager.cc:1397] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:06:54.557549 4766 tablet_bootstrap.cc:492] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Bootstrap starting.
I20260927 04:06:54.557850 4763 tablet_bootstrap.cc:492] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: No bootstrap required, opened a new log
I20260927 04:06:54.557931 4763 ts_tablet_manager.cc:1397] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:06:54.558211 4766 tablet_bootstrap.cc:654] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:54.558555 4766 log.cc:824] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:54.559362 4766 tablet_bootstrap.cc:492] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: No bootstrap required, opened a new log
I20260927 04:06:54.559432 4766 ts_tablet_manager.cc:1397] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:06:54.559823 4762 raft_consensus.cc:348] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.559978 4762 raft_consensus.cc:374] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:54.560019 4762 raft_consensus.cc:729] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba2010a96844bedb420a33b1bab7a13, State: Initialized, Role: FOLLOWER
I20260927 04:06:54.560163 4762 consensus_queue.cc:260] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.560411 4762 ts_tablet_manager.cc:1428] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:06:54.560541 4486 heartbeater.cc:503] Master 127.17.184.254:44979 was elected leader, sending a full tablet report...
I20260927 04:06:54.560832 4763 raft_consensus.cc:348] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.560993 4763 raft_consensus.cc:374] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:54.561028 4763 raft_consensus.cc:729] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcef3846848c4110afac2bba53ea60f4, State: Initialized, Role: FOLLOWER
I20260927 04:06:54.561151 4763 consensus_queue.cc:260] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.561398 4763 ts_tablet_manager.cc:1428] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260927 04:06:54.561439 4748 heartbeater.cc:503] Master 127.17.184.254:44979 was elected leader, sending a full tablet report...
I20260927 04:06:54.562247 4766 raft_consensus.cc:348] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.562409 4766 raft_consensus.cc:374] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:54.562448 4766 raft_consensus.cc:729] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84058544041a4171bdfb4af79cfa6748, State: Initialized, Role: FOLLOWER
I20260927 04:06:54.562538 4766 consensus_queue.cc:260] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.562788 4617 heartbeater.cc:503] Master 127.17.184.254:44979 was elected leader, sending a full tablet report...
I20260927 04:06:54.563024 4766 ts_tablet_manager.cc:1428] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260927 04:06:54.646225 4618 tablet.cc:2369] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:06:54.745139 4487 tablet.cc:2369] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 04:06:54.776880 4749 tablet.cc:2369] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:06:54.878582 4770 raft_consensus.cc:482] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:54.878744 4770 raft_consensus.cc:504] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.879149 4770 leader_election.cc:288] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcef3846848c4110afac2bba53ea60f4 (127.17.184.195:46249), 4ba2010a96844bedb420a33b1bab7a13 (127.17.184.193:38133)
I20260927 04:06:54.881987 4619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.879386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:38133 (local address 127.17.184.194:39839)
0927 04:06:54.879453 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.879467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:54.879531 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:54.879736 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:54.879737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:54.879738 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:54.879813 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.879814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.880443 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.880444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.881194 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.881196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.881733 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.881734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.881809 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.881812 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:54.881878 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 04:06:54.881919 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 04:06:54.881987 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.879545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:39839 (local address 127.17.184.193:38133)
0927 04:06:54.879592 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.879594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.879603 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:54.879637 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.879638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.879647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.879732 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:54.879884 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.880360 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.881287 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.881628 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.881729 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.881738 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.881885 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.881921 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:06:54.882365 4771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.879396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:46249 (local address 127.17.184.194:51829)
0927 04:06:54.879604 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.879615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:54.879689 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:54.879852 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:54.879853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:54.879855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:54.879934 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.879938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.880642 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.880643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.881449 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:54.881451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.882132 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.882133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:54.882201 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.882205 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:54.882260 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:06:54.882291 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":72,"thread_start_us":67,"threads_started":1}
I20260927 04:06:54.882331 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ffe6e9bffd417d92a9fc3981db2fd9" candidate_uuid: "84058544041a4171bdfb4af79cfa6748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba2010a96844bedb420a33b1bab7a13" is_pre_election: true
I20260927 04:06:54.882400 4759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.879480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:51829 (local address 127.17.184.195:46249)
0927 04:06:54.879594 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.879597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.879696 (+ 99us) server_negotiation.cc:407] Connection header received
0927 04:06:54.879753 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.879754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.879764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.879850 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:54.880040 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.880546 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.881557 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.881968 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.882108 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.882119 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.882285 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.882329 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 04:06:54.882521 4440 raft_consensus.cc:2454] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84058544041a4171bdfb4af79cfa6748 in term 0.
I20260927 04:06:54.882788 4505 leader_election.cc:302] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba2010a96844bedb420a33b1bab7a13, 84058544041a4171bdfb4af79cfa6748; no voters:
I20260927 04:06:54.882784 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ffe6e9bffd417d92a9fc3981db2fd9" candidate_uuid: "84058544041a4171bdfb4af79cfa6748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcef3846848c4110afac2bba53ea60f4" is_pre_election: true
I20260927 04:06:54.882946 4702 raft_consensus.cc:2454] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84058544041a4171bdfb4af79cfa6748 in term 0.
I20260927 04:06:54.882932 4770 raft_consensus.cc:2781] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:06:54.882992 4770 raft_consensus.cc:482] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:06:54.883034 4770 raft_consensus.cc:3037] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:54.883827 4770 raft_consensus.cc:504] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.883978 4770 leader_election.cc:288] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [CANDIDATE]: Term 1 election: Requested vote from peers bcef3846848c4110afac2bba53ea60f4 (127.17.184.195:46249), 4ba2010a96844bedb420a33b1bab7a13 (127.17.184.193:38133)
I20260927 04:06:54.884152 4440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ffe6e9bffd417d92a9fc3981db2fd9" candidate_uuid: "84058544041a4171bdfb4af79cfa6748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba2010a96844bedb420a33b1bab7a13"
I20260927 04:06:54.884270 4702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75ffe6e9bffd417d92a9fc3981db2fd9" candidate_uuid: "84058544041a4171bdfb4af79cfa6748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcef3846848c4110afac2bba53ea60f4"
I20260927 04:06:54.884268 4440 raft_consensus.cc:3037] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:54.884346 4702 raft_consensus.cc:3037] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:54.885283 4440 raft_consensus.cc:2454] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84058544041a4171bdfb4af79cfa6748 in term 1.
I20260927 04:06:54.885282 4702 raft_consensus.cc:2454] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84058544041a4171bdfb4af79cfa6748 in term 1.
I20260927 04:06:54.885474 4505 leader_election.cc:302] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba2010a96844bedb420a33b1bab7a13, 84058544041a4171bdfb4af79cfa6748; no voters:
I20260927 04:06:54.885609 4770 raft_consensus.cc:2781] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:54.885782 4770 raft_consensus.cc:686] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 1 LEADER]: Becoming Leader. State: Replica: 84058544041a4171bdfb4af79cfa6748, State: Running, Role: LEADER
I20260927 04:06:54.885890 4770 consensus_queue.cc:237] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } }
I20260927 04:06:54.886737 4298 catalog_manager.cc:5495] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84058544041a4171bdfb4af79cfa6748 (127.17.184.194). New cstate: current_term: 1 leader_uuid: "84058544041a4171bdfb4af79cfa6748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84058544041a4171bdfb4af79cfa6748" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43165 } health_report { overall_health: HEALTHY } } }
I20260927 04:06:54.904146 4761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.901737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.17.184.194:43165)
0927 04:06:54.901791 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:54.901793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:54.901805 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:06:54.901877 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:54.901878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:54.901896 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:54.901973 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:06:54.902126 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.902592 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.903363 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:54.903555 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:54.903640 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:54.903775 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:06:54.903888 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:06:54.903999 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:54.904035 (+ 36us) server_negotiation.cc:299] Negotiation successful
0927 04:06:54.904046 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 04:06:54.905905 4702 raft_consensus.cc:1261] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 1 FOLLOWER]: Refusing update from remote peer 84058544041a4171bdfb4af79cfa6748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:54.905947 4440 raft_consensus.cc:1261] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 4ba2010a96844bedb420a33b1bab7a13 [term 1 FOLLOWER]: Refusing update from remote peer 84058544041a4171bdfb4af79cfa6748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:54.906307 4772 consensus_queue.cc:1035] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcef3846848c4110afac2bba53ea60f4" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 46249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:54.906448 4770 consensus_queue.cc:1035] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ba2010a96844bedb420a33b1bab7a13" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:54.908771 4775 mvcc.cc:205] Tried to move back new op lower bound from 7333814333052002304 to 7333814332973309952. Current Snapshot: MvccSnapshot[applied={T|T < 7333814333052002304}]
I20260927 04:06:54.909430 4777 mvcc.cc:205] Tried to move back new op lower bound from 7333814333052002304 to 7333814332973309952. Current Snapshot: MvccSnapshot[applied={T|T < 7333814333052002304}]
I20260927 04:06:54.911507 4298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:54.911603 4298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:54.912352 4298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abc294b0821c4538bcb9b981c017e98b: Sending DeleteTablet for 3 replicas of tablet 75ffe6e9bffd417d92a9fc3981db2fd9
I20260927 04:06:54.912690 4420 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ffe6e9bffd417d92a9fc3981db2fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:54 UTC) from {username='slave'} at 127.0.0.1:46974
I20260927 04:06:54.912712 4682 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ffe6e9bffd417d92a9fc3981db2fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:54 UTC) from {username='slave'} at 127.0.0.1:60942
I20260927 04:06:54.912896 4551 tablet_service.cc:1514] Processing DeleteTablet for tablet 75ffe6e9bffd417d92a9fc3981db2fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:54 UTC) from {username='slave'} at 127.0.0.1:52618
I20260927 04:06:54.913110 4787 tablet_replica.cc:330] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: stopping tablet replica
I20260927 04:06:54.913188 4785 tablet_replica.cc:330] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: stopping tablet replica
I20260927 04:06:54.913291 4785 raft_consensus.cc:2229] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:54.913314 4787 raft_consensus.cc:2229] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:06:54.913443 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 4358
I20260927 04:06:54.913398 4785 raft_consensus.cc:2258] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:54.914531 4787 raft_consensus.cc:2258] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:54.915086 4787 ts_tablet_manager.cc:1905] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:54.916046 4785 ts_tablet_manager.cc:1905] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:54.916985 4787 ts_tablet_manager.cc:1918] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:54.917059 4787 log.cc:1196] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/75ffe6e9bffd417d92a9fc3981db2fd9
I20260927 04:06:54.917330 4787 ts_tablet_manager.cc:1939] T 75ffe6e9bffd417d92a9fc3981db2fd9 P 84058544041a4171bdfb4af79cfa6748: Deleting consensus metadata
I20260927 04:06:54.917724 4785 ts_tablet_manager.cc:1918] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:54.917793 4785 log.cc:1196] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/75ffe6e9bffd417d92a9fc3981db2fd9
I20260927 04:06:54.917884 4282 catalog_manager.cc:4841] TS 84058544041a4171bdfb4af79cfa6748 (127.17.184.194:43165): tablet 75ffe6e9bffd417d92a9fc3981db2fd9 (table test-table [id=8f8c1adae829486f8a37463567a12c6a]) successfully deleted
I20260927 04:06:54.918047 4785 ts_tablet_manager.cc:1939] T 75ffe6e9bffd417d92a9fc3981db2fd9 P bcef3846848c4110afac2bba53ea60f4: Deleting consensus metadata
I20260927 04:06:54.918603 4284 catalog_manager.cc:4841] TS bcef3846848c4110afac2bba53ea60f4 (127.17.184.195:46249): tablet 75ffe6e9bffd417d92a9fc3981db2fd9 (table test-table [id=8f8c1adae829486f8a37463567a12c6a]) successfully deleted
W20260927 04:06:54.921373 4283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:38133 (error 108)
I20260927 04:06:54.921670 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 4489
I20260927 04:06:54.922251 4756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:54.921765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:38133 (local address 127.0.0.1:46976)
0927 04:06:54.922165 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:54.922217 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:38133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378}
W20260927 04:06:54.922421 4283 catalog_manager.cc:4584] TS 4ba2010a96844bedb420a33b1bab7a13 (127.17.184.193:38133): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75ffe6e9bffd417d92a9fc3981db2fd9: Network error: Client connection negotiation failed: client connection to 127.17.184.193:38133: connect: Connection refused (error 111)
I20260927 04:06:54.926375 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 4620
I20260927 04:06:54.930785 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 4267
2026-09-27T04:06:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1020 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:06:54 dist-test-slave-zqrn krb5kdc[4791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:06:54 dist-test-slave-zqrn krb5kdc[4791](info): set up 2 sockets
Sep 27 04:06:54 dist-test-slave-zqrn krb5kdc[4791](info): commencing operation
krb5kdc: starting...
W20260927 04:06:55.962885 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:06:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:06:56.090175 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:46791
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:56.175879 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:56.176093 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:56.176118 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:56.177228 4804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:56.177266 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:56.177280 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:56.177292 4804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:56.177304 4804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:56.178709 4804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:46791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:56.178972 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:56.179167 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:56.184015 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:56.184096 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:56.184015 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:56.184305 4804 server_base.cc:1034] running on GCE node
I20260927 04:06:56.184621 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:56.184937 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:56.186122 4804 hybrid_clock.cc:648] HybridClock initialized: now 1790482016186095 us; error 42 us; skew 500 ppm
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:56.188689 4804 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:06:56.189253 4804 webserver.cc:460] Webserver started at http://127.17.184.254:36895/ using document root <none> and password file <none>
I20260927 04:06:56.189520 4804 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:56.189601 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:56.189764 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:56.191079 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "b9bc228728014bb2a342b162637e41ef"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "955bca1a04d2328ec1eeecd068a01d27"
server_key_iv: "2801cb9d320689d07932a2a9e1782418"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.191491 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "b9bc228728014bb2a342b162637e41ef"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "955bca1a04d2328ec1eeecd068a01d27"
server_key_iv: "2801cb9d320689d07932a2a9e1782418"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.193046 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:56.193938 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.194131 4804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:56.194235 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "b9bc228728014bb2a342b162637e41ef"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "955bca1a04d2328ec1eeecd068a01d27"
server_key_iv: "2801cb9d320689d07932a2a9e1782418"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.194317 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:56.209817 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:56.211769 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:56.211951 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:56.215974 4804 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:46791
I20260927 04:06:56.216012 4871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:46791 every 8 connection(s)
I20260927 04:06:56.216372 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:56.217228 4872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:56.219466 4872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef: Bootstrap starting.
I20260927 04:06:56.219977 4872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:56.220217 4872 log.cc:824] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:56.220845 4872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef: No bootstrap required, opened a new log
I20260927 04:06:56.222343 4872 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bc228728014bb2a342b162637e41ef" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46791 } }
I20260927 04:06:56.222482 4872 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:56.222505 4872 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bc228728014bb2a342b162637e41ef, State: Initialized, Role: FOLLOWER
I20260927 04:06:56.222604 4872 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bc228728014bb2a342b162637e41ef" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46791 } }
I20260927 04:06:56.222698 4872 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:56.222731 4872 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:56.222785 4872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:56.223344 4872 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bc228728014bb2a342b162637e41ef" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46791 } }
I20260927 04:06:56.223464 4872 leader_election.cc:302] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9bc228728014bb2a342b162637e41ef; no voters:
I20260927 04:06:56.223642 4872 leader_election.cc:288] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:06:56.223698 4875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:56.223861 4875 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [term 1 LEADER]: Becoming Leader. State: Replica: b9bc228728014bb2a342b162637e41ef, State: Running, Role: LEADER
I20260927 04:06:56.223898 4872 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:56.223995 4875 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bc228728014bb2a342b162637e41ef" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46791 } }
I20260927 04:06:56.224395 4877 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9bc228728014bb2a342b162637e41ef. Latest consensus state: current_term: 1 leader_uuid: "b9bc228728014bb2a342b162637e41ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bc228728014bb2a342b162637e41ef" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46791 } } }
I20260927 04:06:56.224524 4876 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9bc228728014bb2a342b162637e41ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bc228728014bb2a342b162637e41ef" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 46791 } } }
I20260927 04:06:56.224555 4877 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:56.224640 4876 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:56.224958 4884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:56.225500 4884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:06:56.225741 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 4804
I20260927 04:06:56.225852 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:06:56.226068 18147 external_mini_cluster.cc:1428] Setting key bf71e0302ef818a4ebc4c6fa428a370d
I20260927 04:06:56.227145 4884 catalog_manager.cc:1347] Generated new cluster ID: 45ca7976b56c44d2884dc45fb8fd7f1d
I20260927 04:06:56.227200 4884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:56.242268 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.226605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60792 (local address 127.17.184.254:46791)
0927 04:06:56.231115 (+ 4510us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.231124 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.231161 (+ 37us) server_negotiation.cc:407] Connection header received
0927 04:06:56.231278 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.231285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.231338 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.231476 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:56.232422 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.233833 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.234975 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.235592 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.235698 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.238551 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:56.238556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:56.238556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:56.238571 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:56.240589 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.240979 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.240982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.240983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.241097 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.241290 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.241291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.241291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.241624 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:56.241776 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.241886 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.242025 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4459,"thread_start_us":91,"threads_started":1}
I20260927 04:06:56.243830 4884 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:06:56.244378 4884 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:56.251633 4884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef: Generated new TSK 0
I20260927 04:06:56.251868 4884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.184.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
Entry for principal kudu/127.17.184.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab.
I20260927 04:06:56.286322 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46791
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:56.368114 4898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:56.368335 4898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:56.368374 4898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:56.369504 4898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:56.369568 4898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:56.369625 4898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:06:56.371136 4898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.193.keytab
--principal=kudu/127.17.184.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:56.371389 4898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:56.371621 4898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:56.377002 4903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:56.376974 4904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:56.376953 4906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:56.377271 4898 server_base.cc:1034] running on GCE node
I20260927 04:06:56.377485 4898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:56.377731 4898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:56.378970 4898 hybrid_clock.cc:648] HybridClock initialized: now 1790482016378955 us; error 30 us; skew 500 ppm
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:56.381819 4898 init.cc:377] Logged in from keytab as kudu/127.17.184.193@KRBTEST.COM (short username kudu)
I20260927 04:06:56.382262 4898 webserver.cc:460] Webserver started at http://127.17.184.193:33409/ using document root <none> and password file <none>
I20260927 04:06:56.382520 4898 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:56.382588 4898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:56.382750 4898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:56.383925 4898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "a91a4a47ebbd11964b78ca09395ded2f"
server_key_iv: "4651059c08748a710a714025c949794a"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.384289 4898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "a91a4a47ebbd11964b78ca09395ded2f"
server_key_iv: "4651059c08748a710a714025c949794a"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.385630 4898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:56.386412 4913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.386607 4898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:06:56.386711 4898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "a91a4a47ebbd11964b78ca09395ded2f"
server_key_iv: "4651059c08748a710a714025c949794a"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.386796 4898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:56.394392 4898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:56.397107 4898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:56.397265 4898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:56.397488 4898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:56.397809 4898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:56.397862 4898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.397905 4898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:56.397931 4898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.403940 4898 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:44487
I20260927 04:06:56.403993 5026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:44487 every 8 connection(s)
I20260927 04:06:56.404310 4898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.193@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:56.411856 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 4898
I20260927 04:06:56.411978 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:06:56.412223 18147 external_mini_cluster.cc:1428] Setting key 8330606dc1973bbc6152e0231377c705
I20260927 04:06:56.414378 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.405035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:37947 (local address 127.17.184.254:46791)
0927 04:06:56.405097 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.405099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.405311 (+ 212us) server_negotiation.cc:407] Connection header received
0927 04:06:56.405821 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.405822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.405833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.405898 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:56.407115 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.407688 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.408710 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.408906 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.408961 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.411256 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:56.411257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:56.411257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:56.411264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:56.412848 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.413275 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.413276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.413277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.413338 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.413632 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.413633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.413633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.413786 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:56.413856 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.414139 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.414238 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:06:56.414422 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.405019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:46791 (local address 127.17.184.193:37947)
0927 04:06:56.405170 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.405213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:56.405733 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:56.405982 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:56.405994 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:56.406609 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:56.406875 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.406882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.407806 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.407807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.408625 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.408627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.408987 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.408988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.409106 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.409735 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:56.409744 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:56.411171 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:56.412932 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:56.412939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:56.412941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:56.413201 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:56.413440 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:56.413440 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:56.413441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:56.413553 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:56.413860 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:56.413866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:56.413998 (+ 132us) client_negotiation.cc:769] Sending connection context
0927 04:06:56.414091 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 04:06:56.414218 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260927 04:06:56.415220 5027 heartbeater.cc:347] Connected to a master server at 127.17.184.254:46791
I20260927 04:06:56.415316 5027 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:56.415514 5027 heartbeater.cc:511] Master 127.17.184.254:46791 requested a full tablet report, sending...
I20260927 04:06:56.416118 4836 ts_manager.cc:194] Registered new tserver with Master: edb2f8c6c4c341e2ac43a2d2bcc3160c (127.17.184.193:44487)
I20260927 04:06:56.416733 4836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.193@KRBTEST.COM'} at 127.17.184.193:37947
WARNING: no policy specified for kudu/127.17.184.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
Entry for principal kudu/127.17.184.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab.
I20260927 04:06:56.438668 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46791
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:56.522121 5032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:56.522337 5032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:56.522362 5032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:56.523640 5032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:56.523684 5032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:56.523713 5032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:06:56.525208 5032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.194.keytab
--principal=kudu/127.17.184.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:56.525408 5032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:56.525592 5032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:56.530787 5037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:56.530743 5040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:56.530791 5032 server_base.cc:1034] running on GCE node
W20260927 04:06:56.530755 5038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:56.531276 5032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:56.531476 5032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:56.532624 5032 hybrid_clock.cc:648] HybridClock initialized: now 1790482016532604 us; error 37 us; skew 500 ppm
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:56.535166 5032 init.cc:377] Logged in from keytab as kudu/127.17.184.194@KRBTEST.COM (short username kudu)
I20260927 04:06:56.535581 5032 webserver.cc:460] Webserver started at http://127.17.184.194:38873/ using document root <none> and password file <none>
I20260927 04:06:56.535799 5032 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:56.535866 5032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:56.536016 5032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:56.537083 5032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "e32c924b7063448e92e5d387640589f8"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "64d043ff105babe5e4c56c4627e5f083"
server_key_iv: "01da89dee8d9b34673da18bcec526ef7"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.537436 5032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "e32c924b7063448e92e5d387640589f8"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "64d043ff105babe5e4c56c4627e5f083"
server_key_iv: "01da89dee8d9b34673da18bcec526ef7"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.538795 5032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:56.539646 5047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.539835 5032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:06:56.539934 5032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "e32c924b7063448e92e5d387640589f8"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "64d043ff105babe5e4c56c4627e5f083"
server_key_iv: "01da89dee8d9b34673da18bcec526ef7"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.540031 5032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:56.557092 5032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:56.559489 5032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:56.559657 5032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:56.559907 5032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:56.560220 5032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:56.560274 5032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.560314 5032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:56.560343 5032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.566133 5032 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:33099
I20260927 04:06:56.566246 5160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:33099 every 8 connection(s)
I20260927 04:06:56.566538 5032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.194@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:56.574151 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 5032
I20260927 04:06:56.574280 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:06:56.574457 18147 external_mini_cluster.cc:1428] Setting key 4efa69d53a7181cfceef466c0dcfdaa9
I20260927 04:06:56.577279 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.567395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:43887 (local address 127.17.184.254:46791)
0927 04:06:56.567491 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.567495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.567732 (+ 237us) server_negotiation.cc:407] Connection header received
0927 04:06:56.568741 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.568743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.568755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.568814 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:56.569453 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.569962 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.571044 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.571253 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.571339 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.573687 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:56.573689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:56.573689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:56.573697 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:56.575594 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.576143 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.576144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.576144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.576204 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.576560 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.576561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.576561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.576694 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:56.576798 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.577102 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.577187 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 04:06:56.577488 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.567356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:46791 (local address 127.17.184.194:43887)
0927 04:06:56.567591 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.567631 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:56.568614 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:56.568884 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:56.568890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:56.569155 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:56.569348 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.569352 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.570067 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.570068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.570887 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.570937 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.571357 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.571358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.571483 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.571981 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:56.571990 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:56.573585 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:56.575675 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:56.575682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:56.575685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:56.576009 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:56.576280 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:56.576281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:56.576281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:56.576432 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:56.576791 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:56.576797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:56.576946 (+ 149us) client_negotiation.cc:769] Sending connection context
0927 04:06:56.577067 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 04:06:56.577236 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260927 04:06:56.578090 5161 heartbeater.cc:347] Connected to a master server at 127.17.184.254:46791
I20260927 04:06:56.578234 5161 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:56.578421 5161 heartbeater.cc:511] Master 127.17.184.254:46791 requested a full tablet report, sending...
I20260927 04:06:56.578855 4836 ts_manager.cc:194] Registered new tserver with Master: e32c924b7063448e92e5d387640589f8 (127.17.184.194:33099)
I20260927 04:06:56.579355 4836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.194@KRBTEST.COM'} at 127.17.184.194:43887
WARNING: no policy specified for kudu/127.17.184.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
Entry for principal kudu/127.17.184.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab.
I20260927 04:06:56.602162 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46791
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:34343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:56.683369 5166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:56.683624 5166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:56.683662 5166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:56.684862 5166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:56.684939 5166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:56.684984 5166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:06:56.686441 5166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:34343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.195.keytab
--principal=kudu/127.17.184.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:46791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:56.686707 5166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:56.686954 5166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:56.692006 5174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:56.692000 5172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:56.692031 5171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:56.692260 5166 server_base.cc:1034] running on GCE node
I20260927 04:06:56.692525 5166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:56.692780 5166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:56.693943 5166 hybrid_clock.cc:648] HybridClock initialized: now 1790482016693931 us; error 30 us; skew 500 ppm
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:56.696470 5166 init.cc:377] Logged in from keytab as kudu/127.17.184.195@KRBTEST.COM (short username kudu)
I20260927 04:06:56.696971 5166 webserver.cc:460] Webserver started at http://127.17.184.195:45555/ using document root <none> and password file <none>
I20260927 04:06:56.697175 5166 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:56.697216 5166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:56.697304 5166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:56.698372 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "10043b8b32804edb962c83ecb1701ef5"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "0209e7833b5e505a2a80bfff5c3650fc"
server_key_iv: "91b3a0813290491e1594dbefd657d660"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.698669 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "10043b8b32804edb962c83ecb1701ef5"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "0209e7833b5e505a2a80bfff5c3650fc"
server_key_iv: "91b3a0813290491e1594dbefd657d660"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.700037 5166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:06:56.700749 5181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.700964 5166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:06:56.701057 5166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "10043b8b32804edb962c83ecb1701ef5"
format_stamp: "Formatted at 2026-09-27 04:06:56 on dist-test-slave-zqrn"
server_key: "0209e7833b5e505a2a80bfff5c3650fc"
server_key_iv: "91b3a0813290491e1594dbefd657d660"
server_key_version: "encryptionkey@0"
I20260927 04:06:56.701122 5166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:56.720264 5166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:56.722452 5166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:56.722571 5166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:56.722759 5166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:56.723131 5166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:56.723166 5166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.723189 5166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:56.723239 5166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:56.729493 5166 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:42263
I20260927 04:06:56.729568 5294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:42263 every 8 connection(s)
I20260927 04:06:56.729852 5166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
Sep 27 04:06:56 dist-test-slave-zqrn krb5kdc[4791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482016, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.195@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:06:56.737391 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 5166
I20260927 04:06:56.737516 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:06:56.737726 18147 external_mini_cluster.cc:1428] Setting key 2823cda911747a7000aa95d5761c7ad6
I20260927 04:06:56.740058 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.730611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:59043 (local address 127.17.184.254:46791)
0927 04:06:56.730669 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.730671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.730844 (+ 173us) server_negotiation.cc:407] Connection header received
0927 04:06:56.731390 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.731391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.731404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.731483 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:56.732430 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.732981 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.734156 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.734373 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.734442 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.736897 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:56.736899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:56.736899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:56.736907 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:56.738851 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.739283 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.739284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.739285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.739341 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.739525 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.739525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.739526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.739634 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:56.739690 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.739931 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.739994 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:06:56.740296 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.730526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:46791 (local address 127.17.184.195:59043)
0927 04:06:56.730692 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.730734 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:56.731275 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:56.731538 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:56.731542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:56.732101 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:56.732311 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.732316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.733106 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.733107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.734055 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.734058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.734497 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.734499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.734617 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.735206 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:06:56.735216 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:56.736784 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:56.738978 (+ 2194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:56.738984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:56.738985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:56.739215 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:56.739390 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:06:56.739391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:06:56.739391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:06:56.739476 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:06:56.739693 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:56.739698 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:56.739818 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 04:06:56.739904 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:06:56.740027 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260927 04:06:56.740801 5295 heartbeater.cc:347] Connected to a master server at 127.17.184.254:46791
I20260927 04:06:56.740913 5295 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:56.741094 5295 heartbeater.cc:511] Master 127.17.184.254:46791 requested a full tablet report, sending...
I20260927 04:06:56.741578 4836 ts_manager.cc:194] Registered new tserver with Master: 10043b8b32804edb962c83ecb1701ef5 (127.17.184.195:42263)
I20260927 04:06:56.741912 4836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.184.195@KRBTEST.COM'} at 127.17.184.195:59043
I20260927 04:06:56.750025 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:06:56.757194 4894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.751358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60794 (local address 127.17.184.254:46791)
0927 04:06:56.751439 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.751441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.751518 (+ 77us) server_negotiation.cc:407] Connection header received
0927 04:06:56.751594 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.751595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.751603 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.751722 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:56.752344 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.752820 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.753542 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.753724 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.753773 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.755041 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:56.755042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:56.755043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:56.755048 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:56.756227 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.756533 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.756534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.756534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.756579 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:56.756751 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:56.756751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:56.756751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:56.756887 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:56.756963 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.757071 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.757128 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:06:56.758713 4836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:06:56.760345 4836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:06:56.766742 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.762624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.17.184.194:33099)
0927 04:06:56.762780 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.762785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.762948 (+ 163us) server_negotiation.cc:407] Connection header received
0927 04:06:56.763036 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.763039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.763056 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.763350 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:56.763431 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.764205 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.765592 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.766311 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.766402 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.766507 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.766609 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.766652 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260927 04:06:56.766731 5306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.762542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33099 (local address 127.0.0.1:52746)
0927 04:06:56.762796 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.762840 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:56.762939 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:56.763184 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:56.763188 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:56.763191 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:56.763271 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.763273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.764309 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.764310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.765513 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.765516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.766397 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.766398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.766502 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.766518 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 04:06:56.766593 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 04:06:56.766650 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":58,"threads_started":1}
I20260927 04:06:56.767107 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.762619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:42263 (local address 127.0.0.1:45868)
0927 04:06:56.763150 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.763170 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:56.763217 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:56.763481 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:56.763482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:56.763484 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:56.763525 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.763526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.764806 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.764807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.765841 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.765842 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.766816 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.766817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.766880 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.766885 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 04:06:56.766971 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:06:56.767029 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":55,"thread_start_us":103,"threads_started":1}
I20260927 04:06:56.767297 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.762880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.17.184.193:44487)
0927 04:06:56.763135 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.763141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.763150 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:56.763187 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.763189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.763203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.763273 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:56.763543 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.764695 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.766135 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.766850 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.767015 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.767074 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.767171 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.767222 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260927 04:06:56.767853 5095 tablet_service.cc:1467] Processing CreateTablet for tablet a457d113f0d84317856718765476e317 (DEFAULT_TABLE table=test-table [id=90599ed412674c1aa6994bdf8a8a308a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:56.768182 5095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a457d113f0d84317856718765476e317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:56.770675 5312 tablet_bootstrap.cc:492] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Bootstrap starting.
I20260927 04:06:56.770776 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.762863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.17.184.195:42263)
0927 04:06:56.763103 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.763107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.763215 (+ 108us) server_negotiation.cc:407] Connection header received
0927 04:06:56.763328 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.763331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.763347 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.763418 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:56.763601 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.764695 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.765939 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.766660 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.766734 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.766780 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.767342 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.770688 (+ 3346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":123,"threads_started":1}
I20260927 04:06:56.767282 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.762617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:44487 (local address 127.0.0.1:54506)
0927 04:06:56.763066 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.763076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:56.763121 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:56.763411 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:56.763411 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:56.763413 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:56.763456 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.763457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.764834 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.764836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.766047 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.766050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.767038 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.767039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.767099 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.767103 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:56.767166 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 04:06:56.767210 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":98,"threads_started":1}
I20260927 04:06:56.771458 5312 tablet_bootstrap.cc:654] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:56.771565 5229 tablet_service.cc:1467] Processing CreateTablet for tablet a457d113f0d84317856718765476e317 (DEFAULT_TABLE table=test-table [id=90599ed412674c1aa6994bdf8a8a308a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:56.771636 4961 tablet_service.cc:1467] Processing CreateTablet for tablet a457d113f0d84317856718765476e317 (DEFAULT_TABLE table=test-table [id=90599ed412674c1aa6994bdf8a8a308a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:56.771899 5229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a457d113f0d84317856718765476e317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:56.772007 4961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a457d113f0d84317856718765476e317. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:56.772015 5312 log.cc:824] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:56.772826 5312 tablet_bootstrap.cc:492] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: No bootstrap required, opened a new log
I20260927 04:06:56.772902 5312 ts_tablet_manager.cc:1397] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:56.774758 5314 tablet_bootstrap.cc:492] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: Bootstrap starting.
I20260927 04:06:56.775024 5312 raft_consensus.cc:348] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.775345 5312 raft_consensus.cc:374] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:56.775352 5315 tablet_bootstrap.cc:492] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c: Bootstrap starting.
I20260927 04:06:56.775458 5312 raft_consensus.cc:729] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32c924b7063448e92e5d387640589f8, State: Initialized, Role: FOLLOWER
I20260927 04:06:56.775602 5314 tablet_bootstrap.cc:654] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:56.775666 5312 consensus_queue.cc:260] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.776005 5312 ts_tablet_manager.cc:1428] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:06:56.775995 5315 tablet_bootstrap.cc:654] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:56.776154 5161 heartbeater.cc:503] Master 127.17.184.254:46791 was elected leader, sending a full tablet report...
I20260927 04:06:56.776206 5314 log.cc:824] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:56.776573 5315 log.cc:824] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:56.776999 5314 tablet_bootstrap.cc:492] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: No bootstrap required, opened a new log
I20260927 04:06:56.777086 5314 ts_tablet_manager.cc:1397] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:56.777340 5315 tablet_bootstrap.cc:492] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c: No bootstrap required, opened a new log
I20260927 04:06:56.777406 5315 ts_tablet_manager.cc:1397] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 04:06:56.779042 5314 raft_consensus.cc:348] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.779212 5314 raft_consensus.cc:374] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:56.779245 5314 raft_consensus.cc:729] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10043b8b32804edb962c83ecb1701ef5, State: Initialized, Role: FOLLOWER
I20260927 04:06:56.779357 5314 consensus_queue.cc:260] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.779304 5315 raft_consensus.cc:348] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.779457 5315 raft_consensus.cc:374] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:56.779506 5315 raft_consensus.cc:729] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb2f8c6c4c341e2ac43a2d2bcc3160c, State: Initialized, Role: FOLLOWER
I20260927 04:06:56.779633 5314 ts_tablet_manager.cc:1428] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 04:06:56.779651 5315 consensus_queue.cc:260] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.779767 5295 heartbeater.cc:503] Master 127.17.184.254:46791 was elected leader, sending a full tablet report...
I20260927 04:06:56.780005 5315 ts_tablet_manager.cc:1428] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:06:56.780014 5027 heartbeater.cc:503] Master 127.17.184.254:46791 was elected leader, sending a full tablet report...
W20260927 04:06:56.817193 5162 tablet.cc:2369] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:06:56.842473 5320 raft_consensus.cc:482] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:56.842602 5320 raft_consensus.cc:504] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.842994 5320 leader_election.cc:288] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e32c924b7063448e92e5d387640589f8 (127.17.184.194:33099), 10043b8b32804edb962c83ecb1701ef5 (127.17.184.195:42263)
I20260927 04:06:56.847046 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.843210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:52215 (local address 127.17.184.195:42263)
0927 04:06:56.843261 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.843264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.843419 (+ 155us) server_negotiation.cc:407] Connection header received
0927 04:06:56.843496 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.843497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.843507 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.843627 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:56.843771 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.844507 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.846053 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.846706 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.846772 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.846779 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.846929 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.846971 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 04:06:56.847046 5307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.843380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:39647 (local address 127.17.184.194:33099)
0927 04:06:56.843474 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.843476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.843485 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:56.843508 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.843509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.843521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.843587 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:56.843771 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.844517 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.846053 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.846650 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.846740 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.846752 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.846929 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.846979 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 04:06:56.847108 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.843172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:33099 (local address 127.17.184.193:39647)
0927 04:06:56.843227 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.843238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:56.843284 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:56.843592 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:56.843592 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:56.843594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:56.843699 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.843700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.844613 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.844613 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.845944 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.845948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.846754 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.846755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.846832 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.846836 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:56.846925 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 04:06:56.846985 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":84}
I20260927 04:06:56.847121 5321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.843187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:42263 (local address 127.17.184.193:52215)
0927 04:06:56.843341 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.843352 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:56.843416 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:56.843640 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:56.843641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:56.843642 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:56.843698 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.843700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.844613 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.844614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.845945 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:56.845948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.846774 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.846775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:56.846834 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.846836 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:06:56.846925 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 04:06:56.846984 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":67,"thread_start_us":106,"threads_started":1}
I20260927 04:06:56.847499 5249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a457d113f0d84317856718765476e317" candidate_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10043b8b32804edb962c83ecb1701ef5" is_pre_election: true
I20260927 04:06:56.847497 5115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a457d113f0d84317856718765476e317" candidate_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32c924b7063448e92e5d387640589f8" is_pre_election: true
I20260927 04:06:56.847657 5115 raft_consensus.cc:2454] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edb2f8c6c4c341e2ac43a2d2bcc3160c in term 0.
I20260927 04:06:56.847666 5249 raft_consensus.cc:2454] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edb2f8c6c4c341e2ac43a2d2bcc3160c in term 0.
I20260927 04:06:56.847899 4915 leader_election.cc:302] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e32c924b7063448e92e5d387640589f8, edb2f8c6c4c341e2ac43a2d2bcc3160c; no voters:
I20260927 04:06:56.848044 5320 raft_consensus.cc:2781] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:06:56.848099 5320 raft_consensus.cc:482] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:06:56.848141 5320 raft_consensus.cc:3037] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:56.848942 5320 raft_consensus.cc:504] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.849140 5320 leader_election.cc:288] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [CANDIDATE]: Term 1 election: Requested vote from peers e32c924b7063448e92e5d387640589f8 (127.17.184.194:33099), 10043b8b32804edb962c83ecb1701ef5 (127.17.184.195:42263)
I20260927 04:06:56.849220 5115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a457d113f0d84317856718765476e317" candidate_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32c924b7063448e92e5d387640589f8"
I20260927 04:06:56.849308 5115 raft_consensus.cc:3037] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:56.849291 5249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a457d113f0d84317856718765476e317" candidate_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10043b8b32804edb962c83ecb1701ef5"
I20260927 04:06:56.849391 5249 raft_consensus.cc:3037] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:56.850116 5115 raft_consensus.cc:2454] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edb2f8c6c4c341e2ac43a2d2bcc3160c in term 1.
I20260927 04:06:56.850183 5249 raft_consensus.cc:2454] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edb2f8c6c4c341e2ac43a2d2bcc3160c in term 1.
I20260927 04:06:56.850299 4915 leader_election.cc:302] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e32c924b7063448e92e5d387640589f8, edb2f8c6c4c341e2ac43a2d2bcc3160c; no voters:
I20260927 04:06:56.850379 5320 raft_consensus.cc:2781] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:56.850546 5320 raft_consensus.cc:686] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [term 1 LEADER]: Becoming Leader. State: Replica: edb2f8c6c4c341e2ac43a2d2bcc3160c, State: Running, Role: LEADER
I20260927 04:06:56.850653 5320 consensus_queue.cc:237] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } }
I20260927 04:06:56.851555 4836 catalog_manager.cc:5495] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c reported cstate change: term changed from 0 to 1, leader changed from <none> to edb2f8c6c4c341e2ac43a2d2bcc3160c (127.17.184.193). New cstate: current_term: 1 leader_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edb2f8c6c4c341e2ac43a2d2bcc3160c" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44487 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 } health_report { overall_health: UNKNOWN } } }
I20260927 04:06:56.859953 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.857391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.17.184.193:44487)
0927 04:06:56.857436 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:56.857438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:56.857463 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:06:56.857541 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:56.857542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:56.857561 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:56.857631 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:06:56.857776 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.858254 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.859164 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:56.859360 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:56.859441 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:56.859612 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:06:56.859714 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:06:56.859808 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:56.859828 (+ 20us) server_negotiation.cc:299] Negotiation successful
0927 04:06:56.859848 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 04:06:56.861732 5115 raft_consensus.cc:1261] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 1 FOLLOWER]: Refusing update from remote peer edb2f8c6c4c341e2ac43a2d2bcc3160c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:56.861732 5249 raft_consensus.cc:1261] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 1 FOLLOWER]: Refusing update from remote peer edb2f8c6c4c341e2ac43a2d2bcc3160c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:56.862088 5320 consensus_queue.cc:1035] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32c924b7063448e92e5d387640589f8" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 33099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:56.862238 5322 consensus_queue.cc:1035] T a457d113f0d84317856718765476e317 P edb2f8c6c4c341e2ac43a2d2bcc3160c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10043b8b32804edb962c83ecb1701ef5" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 42263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:56.866051 5328 mvcc.cc:205] Tried to move back new op lower bound from 7333814341063016448 to 7333814341021011968. Current Snapshot: MvccSnapshot[applied={T|T < 7333814341063016448}]
I20260927 04:06:56.866537 4836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:56.866626 4836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:56.867429 4836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9bc228728014bb2a342b162637e41ef: Sending DeleteTablet for 3 replicas of tablet a457d113f0d84317856718765476e317
I20260927 04:06:56.867760 5229 tablet_service.cc:1514] Processing DeleteTablet for tablet a457d113f0d84317856718765476e317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:56 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:45868
I20260927 04:06:56.867776 5095 tablet_service.cc:1514] Processing DeleteTablet for tablet a457d113f0d84317856718765476e317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:56 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:52746
I20260927 04:06:56.868001 4961 tablet_service.cc:1514] Processing DeleteTablet for tablet a457d113f0d84317856718765476e317 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:56 UTC) from {username='kudu', principal='kudu/127.17.184.254@KRBTEST.COM'} at 127.0.0.1:54506
I20260927 04:06:56.868156 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 4898
I20260927 04:06:56.868677 5335 tablet_replica.cc:330] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: stopping tablet replica
I20260927 04:06:56.868788 5335 raft_consensus.cc:2229] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:56.868887 5335 raft_consensus.cc:2258] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:56.868980 5334 tablet_replica.cc:330] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: stopping tablet replica
I20260927 04:06:56.869086 5334 raft_consensus.cc:2229] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:56.869182 5334 raft_consensus.cc:2258] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:56.869387 5335 ts_tablet_manager.cc:1905] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:56.870929 5335 ts_tablet_manager.cc:1918] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:56.871001 5335 log.cc:1196] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/a457d113f0d84317856718765476e317
I20260927 04:06:56.871258 5335 ts_tablet_manager.cc:1939] T a457d113f0d84317856718765476e317 P e32c924b7063448e92e5d387640589f8: Deleting consensus metadata
I20260927 04:06:56.871860 4821 catalog_manager.cc:4841] TS e32c924b7063448e92e5d387640589f8 (127.17.184.194:33099): tablet a457d113f0d84317856718765476e317 (table test-table [id=90599ed412674c1aa6994bdf8a8a308a]) successfully deleted
W20260927 04:06:56.874190 4820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:44487 (error 108)
I20260927 04:06:56.874617 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 5032
I20260927 04:06:56.874750 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:56.874605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:44487 (local address 127.0.0.1:54522)
0927 04:06:56.874662 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:56.874705 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:44487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 04:06:56.874951 5334 ts_tablet_manager.cc:1905] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 04:06:56.874941 4820 catalog_manager.cc:4584] TS edb2f8c6c4c341e2ac43a2d2bcc3160c (127.17.184.193:44487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a457d113f0d84317856718765476e317: Network error: Client connection negotiation failed: client connection to 127.17.184.193:44487: connect: Connection refused (error 111)
I20260927 04:06:56.876734 5334 ts_tablet_manager.cc:1918] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:56.876823 5334 log.cc:1196] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/a457d113f0d84317856718765476e317
I20260927 04:06:56.877111 5334 ts_tablet_manager.cc:1939] T a457d113f0d84317856718765476e317 P 10043b8b32804edb962c83ecb1701ef5: Deleting consensus metadata
I20260927 04:06:56.877552 4823 catalog_manager.cc:4841] TS 10043b8b32804edb962c83ecb1701ef5 (127.17.184.195:42263): tablet a457d113f0d84317856718765476e317 (table test-table [id=90599ed412674c1aa6994bdf8a8a308a]) successfully deleted
I20260927 04:06:56.880018 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 5166
I20260927 04:06:56.884792 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 4804
2026-09-27T04:06:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1955 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T04:06:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:06:56Z Disabled control of system clock
I20260927 04:06:56.912992 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44817
--webserver_interface=127.17.184.254
--webserver_port=0
--builtin_ntp_servers=127.17.184.212:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:56.997720 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:56.997943 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:56.997963 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:56.999083 5341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:56.999118 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:56.999131 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:56.999145 5341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:56.999156 5341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:57.000412 5341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:57.000620 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:57.000872 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:57.005980 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:57.006029 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:57.006238 5341 server_base.cc:1034] running on GCE node
W20260927 04:06:57.005996 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:57.006565 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:57.006948 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:57.008147 5341 hybrid_clock.cc:648] HybridClock initialized: now 1790482017008122 us; error 39 us; skew 500 ppm
I20260927 04:06:57.009523 5341 webserver.cc:460] Webserver started at http://127.17.184.254:37113/ using document root <none> and password file <none>
I20260927 04:06:57.009847 5341 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:57.009927 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:57.010100 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:57.011368 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "e8f55b08c4e74246a0fb1d5f13134af9"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "3413c7e7a2a8b38bb8785ff77a60b06f"
server_key_iv: "408c8e546b05e7288587d797ae52573d"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.011761 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "e8f55b08c4e74246a0fb1d5f13134af9"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "3413c7e7a2a8b38bb8785ff77a60b06f"
server_key_iv: "408c8e546b05e7288587d797ae52573d"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.013250 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:57.014153 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.014350 5341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:57.014467 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "e8f55b08c4e74246a0fb1d5f13134af9"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "3413c7e7a2a8b38bb8785ff77a60b06f"
server_key_iv: "408c8e546b05e7288587d797ae52573d"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.014547 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:57.025197 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:57.025664 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:57.025841 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:57.030059 5341 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44817
I20260927 04:06:57.030092 5407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44817 every 8 connection(s)
I20260927 04:06:57.030474 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:57.031523 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:57.033833 5408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9: Bootstrap starting.
I20260927 04:06:57.034365 5408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:57.034610 5408 log.cc:824] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:57.035328 5408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9: No bootstrap required, opened a new log
I20260927 04:06:57.037281 5408 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f55b08c4e74246a0fb1d5f13134af9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44817 } }
I20260927 04:06:57.037395 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 5341
I20260927 04:06:57.037441 5408 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:57.037468 5408 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8f55b08c4e74246a0fb1d5f13134af9, State: Initialized, Role: FOLLOWER
I20260927 04:06:57.037528 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:06:57.037670 5408 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f55b08c4e74246a0fb1d5f13134af9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44817 } }
I20260927 04:06:57.037765 5408 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:57.037791 5408 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:57.037846 18147 external_mini_cluster.cc:1428] Setting key 1e39edcd888299a1925275dd504a9a45
I20260927 04:06:57.037868 5408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:57.038664 5408 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f55b08c4e74246a0fb1d5f13134af9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44817 } }
I20260927 04:06:57.038813 5408 leader_election.cc:302] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8f55b08c4e74246a0fb1d5f13134af9; no voters:
I20260927 04:06:57.039057 5408 leader_election.cc:288] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:06:57.039090 5413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:57.039294 5413 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [term 1 LEADER]: Becoming Leader. State: Replica: e8f55b08c4e74246a0fb1d5f13134af9, State: Running, Role: LEADER
I20260927 04:06:57.039438 5408 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:57.039443 5413 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f55b08c4e74246a0fb1d5f13134af9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44817 } }
I20260927 04:06:57.039932 5414 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8f55b08c4e74246a0fb1d5f13134af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f55b08c4e74246a0fb1d5f13134af9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44817 } } }
I20260927 04:06:57.040036 5414 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:57.039959 5415 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8f55b08c4e74246a0fb1d5f13134af9. Latest consensus state: current_term: 1 leader_uuid: "e8f55b08c4e74246a0fb1d5f13134af9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8f55b08c4e74246a0fb1d5f13134af9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44817 } } }
I20260927 04:06:57.040211 5415 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:57.041240 5420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:57.042083 5420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:06:57.044163 5420 catalog_manager.cc:1347] Generated new cluster ID: 2543bb947ee049148ad21c26ac0cacf3
I20260927 04:06:57.044221 5420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:06:57.044623 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.038381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.17.184.254:44817)
0927 04:06:57.038620 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.038628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.038653 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:06:57.038770 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.038777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.038847 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.039028 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:57.039329 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.040970 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.042092 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.042678 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.042817 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.043827 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:57.043831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:57.043831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:57.044042 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:57.044086 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:57.044101 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:57.044173 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.044331 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.044413 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":129,"threads_started":1}
I20260927 04:06:57.065852 5420 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:06:57.066608 5420 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:57.071781 5420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9: Generated new TSK 0
I20260927 04:06:57.072000 5420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:06:57.073944 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.193:0
--local_ip_for_outbound_sockets=127.17.184.193
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44817
--builtin_ntp_servers=127.17.184.212:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:57.157989 5432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:57.158191 5432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:57.158213 5432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:57.159444 5432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:57.159482 5432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:57.159507 5432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.193
I20260927 04:06:57.160825 5432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.184.193
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.193
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:57.161060 5432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:57.161378 5432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:57.166426 5437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:57.166431 5438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:57.166549 5440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:57.166775 5432 server_base.cc:1034] running on GCE node
I20260927 04:06:57.167001 5432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:57.167265 5432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:57.168442 5432 hybrid_clock.cc:648] HybridClock initialized: now 1790482017168419 us; error 37 us; skew 500 ppm
I20260927 04:06:57.169798 5432 webserver.cc:460] Webserver started at http://127.17.184.193:41017/ using document root <none> and password file <none>
I20260927 04:06:57.170125 5432 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:57.170208 5432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:57.170364 5432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:57.171495 5432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/data/instance:
uuid: "019243d5ca2e4325b1dcf474d32b3055"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "dc96a19ece4e70c8b94563ab88c713ab"
server_key_iv: "a5d06b56998a0191a2637cd6504e7364"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.171875 5432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance:
uuid: "019243d5ca2e4325b1dcf474d32b3055"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "dc96a19ece4e70c8b94563ab88c713ab"
server_key_iv: "a5d06b56998a0191a2637cd6504e7364"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.173290 5432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:06:57.174178 5446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.174396 5432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:57.174505 5432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/wal
uuid: "019243d5ca2e4325b1dcf474d32b3055"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "dc96a19ece4e70c8b94563ab88c713ab"
server_key_iv: "a5d06b56998a0191a2637cd6504e7364"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.174582 5432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:57.193754 5432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:57.194074 5432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:57.194231 5432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:57.194468 5432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:57.194844 5432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:57.194900 5432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.194965 5432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:57.194991 5432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.201045 5432 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.193:44185
I20260927 04:06:57.201089 5559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.193:44185 every 8 connection(s)
I20260927 04:06:57.201479 5432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/data/info.pb
I20260927 04:06:57.206961 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.202276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:46053 (local address 127.17.184.254:44817)
0927 04:06:57.202375 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.202377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.202588 (+ 211us) server_negotiation.cc:407] Connection header received
0927 04:06:57.203184 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.203185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.203197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.203245 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:57.203906 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.204445 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.205595 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.205798 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.205867 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.206672 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:57.206673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:57.206673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:57.206678 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:57.206692 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:57.206696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:57.206742 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.206828 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.206859 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 04:06:57.207185 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.202267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44817 (local address 127.17.184.193:46053)
0927 04:06:57.202454 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.202494 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:57.203060 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:57.203299 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:57.203302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:57.203310 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:57.203803 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.203807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.204531 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.204532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.205471 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.205473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.205886 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.205886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.205993 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.206539 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:57.206540 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:57.206569 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:57.206591 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:57.206754 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:57.206757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:57.206760 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:57.206811 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 04:06:57.206886 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260927 04:06:57.207794 5560 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44817
I20260927 04:06:57.207885 5560 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:57.208045 5560 heartbeater.cc:511] Master 127.17.184.254:44817 requested a full tablet report, sending...
I20260927 04:06:57.208528 5372 ts_manager.cc:194] Registered new tserver with Master: 019243d5ca2e4325b1dcf474d32b3055 (127.17.184.193:44185)
I20260927 04:06:57.209151 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 5432
I20260927 04:06:57.209215 5372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.193:46053
I20260927 04:06:57.209272 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-0/wal/instance
I20260927 04:06:57.209482 18147 external_mini_cluster.cc:1428] Setting key f6bc8bb4e4645ae2936f4981a2ed3981
I20260927 04:06:57.210460 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.194:0
--local_ip_for_outbound_sockets=127.17.184.194
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44817
--builtin_ntp_servers=127.17.184.212:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:57.293531 5563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:57.293826 5563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:57.293870 5563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:57.295284 5563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:57.295351 5563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:57.295403 5563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.194
I20260927 04:06:57.296968 5563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.184.194
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.194
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:57.297288 5563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:57.297652 5563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:57.303138 5568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:57.303156 5571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:57.303164 5569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:57.303395 5563 server_base.cc:1034] running on GCE node
I20260927 04:06:57.303630 5563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:57.303886 5563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:57.305073 5563 hybrid_clock.cc:648] HybridClock initialized: now 1790482017305058 us; error 29 us; skew 500 ppm
I20260927 04:06:57.306452 5563 webserver.cc:460] Webserver started at http://127.17.184.194:34033/ using document root <none> and password file <none>
I20260927 04:06:57.306754 5563 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:57.306833 5563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:57.307077 5563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:57.308246 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/data/instance:
uuid: "c3bdfc14dcc44c989afea441acfca302"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "d0437f6906b0416091051adce6adfbca"
server_key_iv: "5607b0df5ec16758b7992c46e487f058"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.308620 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance:
uuid: "c3bdfc14dcc44c989afea441acfca302"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "d0437f6906b0416091051adce6adfbca"
server_key_iv: "5607b0df5ec16758b7992c46e487f058"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.309964 5563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:06:57.310828 5577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.311007 5563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:57.311115 5563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/wal
uuid: "c3bdfc14dcc44c989afea441acfca302"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "d0437f6906b0416091051adce6adfbca"
server_key_iv: "5607b0df5ec16758b7992c46e487f058"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.311209 5563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:57.338423 5563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:57.338776 5563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:57.338958 5563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:57.339218 5563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:57.339570 5563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:57.339630 5563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.339679 5563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:57.339715 5563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.345647 5563 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.194:43855
I20260927 04:06:57.345705 5690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.194:43855 every 8 connection(s)
I20260927 04:06:57.346035 5563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/data/info.pb
I20260927 04:06:57.351408 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.346853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.194:51179 (local address 127.17.184.254:44817)
0927 04:06:57.346928 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.346930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.347118 (+ 188us) server_negotiation.cc:407] Connection header received
0927 04:06:57.347619 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.347620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.347630 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.347692 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:57.348363 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.348895 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.349949 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.350165 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.350230 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.351157 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:57.351158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:57.351159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:57.351164 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:57.351182 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:57.351185 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:57.351238 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.351311 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.351342 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 04:06:57.351681 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.346767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44817 (local address 127.17.184.194:51179)
0927 04:06:57.347004 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.347050 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:57.347526 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:57.347744 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:57.347747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:57.347756 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:57.348260 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.348265 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.348984 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.348985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.349884 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.349887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.350251 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.350252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.350361 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.350989 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:57.350990 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:57.351020 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:57.351045 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:57.351252 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:57.351256 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:57.351258 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:06:57.351313 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 04:06:57.351400 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260927 04:06:57.352236 5691 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44817
I20260927 04:06:57.352321 5691 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:57.352538 5691 heartbeater.cc:511] Master 127.17.184.254:44817 requested a full tablet report, sending...
I20260927 04:06:57.352972 5372 ts_manager.cc:194] Registered new tserver with Master: c3bdfc14dcc44c989afea441acfca302 (127.17.184.194:43855)
I20260927 04:06:57.353360 5372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.194:51179
I20260927 04:06:57.355774 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 5563
I20260927 04:06:57.355903 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/wal/instance
I20260927 04:06:57.356109 18147 external_mini_cluster.cc:1428] Setting key fa6955432c9a6b4abb2f30f6cc87d1e0
I20260927 04:06:57.357084 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.184.195:0
--local_ip_for_outbound_sockets=127.17.184.195
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44817
--builtin_ntp_servers=127.17.184.212:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 04:06:57.437330 5694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:57.437541 5694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:57.437563 5694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:57.438721 5694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:57.438761 5694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:57.438786 5694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.184.195
I20260927 04:06:57.440318 5694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.184.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.184.195
--webserver_port=0
--tserver_master_addrs=127.17.184.254:44817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.184.195
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:57.440595 5694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:57.440905 5694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:57.445956 5700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:57.445993 5699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:57.445982 5702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:57.446046 5694 server_base.cc:1034] running on GCE node
I20260927 04:06:57.446326 5694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:57.446657 5694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:57.447829 5694 hybrid_clock.cc:648] HybridClock initialized: now 1790482017447796 us; error 50 us; skew 500 ppm
I20260927 04:06:57.449306 5694 webserver.cc:460] Webserver started at http://127.17.184.195:41967/ using document root <none> and password file <none>
I20260927 04:06:57.449589 5694 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:57.449641 5694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:57.449869 5694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:57.451104 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/data/instance:
uuid: "28afd56259ab4f98976833dd559fba3e"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "fa317d7d7e549433d74168d4023a85de"
server_key_iv: "6a8cf82a65e3e22e71dfbd50fd262ebc"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.451478 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance:
uuid: "28afd56259ab4f98976833dd559fba3e"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "fa317d7d7e549433d74168d4023a85de"
server_key_iv: "6a8cf82a65e3e22e71dfbd50fd262ebc"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.452808 5694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:06:57.453579 5708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.453743 5694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260927 04:06:57.453852 5694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/wal
uuid: "28afd56259ab4f98976833dd559fba3e"
format_stamp: "Formatted at 2026-09-27 04:06:57 on dist-test-slave-zqrn"
server_key: "fa317d7d7e549433d74168d4023a85de"
server_key_iv: "6a8cf82a65e3e22e71dfbd50fd262ebc"
server_key_version: "encryptionkey@0"
I20260927 04:06:57.453933 5694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:57.467808 5694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:57.468183 5694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:57.468356 5694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:57.468626 5694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 04:06:57.468981 5694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 04:06:57.469041 5694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.469080 5694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 04:06:57.469133 5694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:57.475744 5694 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.195:44249
I20260927 04:06:57.475783 5821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.195:44249 every 8 connection(s)
I20260927 04:06:57.476138 5694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/data/info.pb
I20260927 04:06:57.482080 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.476905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.195:33991 (local address 127.17.184.254:44817)
0927 04:06:57.477001 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.477004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.477272 (+ 268us) server_negotiation.cc:407] Connection header received
0927 04:06:57.477797 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.477799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.477810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.477868 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:57.478611 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.479233 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.480338 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.480537 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.480608 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.481738 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:57.481739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:57.481740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:57.481747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:57.481763 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:57.481767 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:57.481829 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.481983 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.482013 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 04:06:57.482080 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 5694
I20260927 04:06:57.482224 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/wal/instance
I20260927 04:06:57.482369 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.476903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:44817 (local address 127.17.184.195:33991)
0927 04:06:57.477113 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.477168 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:57.477717 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:57.477926 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:57.477929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:57.477937 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:06:57.478490 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.478495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.479324 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.479325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.480219 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.480221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.480624 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.480624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.480755 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.481366 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 04:06:57.481366 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:06:57.481396 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 04:06:57.481424 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:06:57.481875 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:06:57.481882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:06:57.481886 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:57.481972 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 04:06:57.482102 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260927 04:06:57.482443 18147 external_mini_cluster.cc:1428] Setting key d01b5757547ebe19fd6b42fe2810aff4
I20260927 04:06:57.482928 5822 heartbeater.cc:347] Connected to a master server at 127.17.184.254:44817
I20260927 04:06:57.483034 5822 heartbeater.cc:464] Registering TS with master...
I20260927 04:06:57.483258 5822 heartbeater.cc:511] Master 127.17.184.254:44817 requested a full tablet report, sending...
I20260927 04:06:57.483758 5372 ts_manager.cc:194] Registered new tserver with Master: 28afd56259ab4f98976833dd559fba3e (127.17.184.195:44249)
I20260927 04:06:57.484139 5372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.184.195:33991
I20260927 04:06:57.494619 18147 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 04:06:57.499156 5412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.496073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.17.184.254:44817)
0927 04:06:57.496150 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.496153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.496226 (+ 73us) server_negotiation.cc:407] Connection header received
0927 04:06:57.496295 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.496296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.496309 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.496381 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:57.496602 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.497094 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.497862 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.498060 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.498134 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.498845 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:57.498846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:57.498846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:06:57.498851 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:57.498873 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:06:57.498877 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:57.498987 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.499042 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.499066 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 04:06:57.500790 5372 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 04:06:57.502260 5372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 04:06:57.508781 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.504704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50862 (local address 127.17.184.194:43855)
0927 04:06:57.504912 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.504916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.504926 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:06:57.504967 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.504969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.504987 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.505073 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:57.505312 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.506135 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.507608 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.508339 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.508422 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.508516 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.508638 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.508696 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260927 04:06:57.508795 5831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.504495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:43855 (local address 127.0.0.1:50862)
0927 04:06:57.504673 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.504715 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:57.504785 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:57.505156 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:57.505160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:57.505162 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:57.505227 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.505229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.506219 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.506220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.507529 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.507531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.508434 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.508435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.508520 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.508535 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 04:06:57.508624 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 04:06:57.508682 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":69,"thread_start_us":55,"threads_started":1}
I20260927 04:06:57.508849 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.504614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.17.184.195:44249)
0927 04:06:57.504799 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.504804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.505050 (+ 246us) server_negotiation.cc:407] Connection header received
0927 04:06:57.505171 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.505174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.505190 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.505268 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:57.505442 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.506184 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.507724 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.508407 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.508467 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.508534 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.508722 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.508758 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20260927 04:06:57.509071 5833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.504526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44249 (local address 127.0.0.1:43090)
0927 04:06:57.504940 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.504972 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:57.505056 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:57.505333 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:57.505334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:57.505335 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:57.505370 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.505371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.506286 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.506287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.507633 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.507636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.508566 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.508567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.508633 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.508636 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:57.508962 (+ 326us) client_negotiation.cc:240] Negotiation successful
0927 04:06:57.509013 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":33,"thread_start_us":88,"threads_started":1}
I20260927 04:06:57.509843 5625 tablet_service.cc:1467] Processing CreateTablet for tablet a9fd906752b94c9095732c89a7282643 (DEFAULT_TABLE table=test-table [id=ac8e04f7935246699b03be659a4aeea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:57.509843 5756 tablet_service.cc:1467] Processing CreateTablet for tablet a9fd906752b94c9095732c89a7282643 (DEFAULT_TABLE table=test-table [id=ac8e04f7935246699b03be659a4aeea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:57.510075 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.504922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.17.184.193:44185)
0927 04:06:57.505141 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.505145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.506330 (+ 1185us) server_negotiation.cc:407] Connection header received
0927 04:06:57.506404 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.506406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.506427 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.506520 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:57.506640 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.507797 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.508806 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.509660 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.509770 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.509855 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.509927 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.509974 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":85,"threads_started":1}
I20260927 04:06:57.510215 5756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fd906752b94c9095732c89a7282643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:57.510215 5625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fd906752b94c9095732c89a7282643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:57.510272 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.504725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:44185 (local address 127.0.0.1:38352)
0927 04:06:57.506267 (+ 1542us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.506277 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:57.506328 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:57.506532 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:57.506533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:57.506535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:57.506570 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.506571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.507893 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.507894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.508708 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.508710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.509783 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.509784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.509847 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.509851 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 04:06:57.509912 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 04:06:57.509955 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1529,"thread_start_us":65,"threads_started":1}
I20260927 04:06:57.511031 5494 tablet_service.cc:1467] Processing CreateTablet for tablet a9fd906752b94c9095732c89a7282643 (DEFAULT_TABLE table=test-table [id=ac8e04f7935246699b03be659a4aeea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 04:06:57.511372 5494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fd906752b94c9095732c89a7282643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:57.513811 5837 tablet_bootstrap.cc:492] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: Bootstrap starting.
I20260927 04:06:57.514281 5838 tablet_bootstrap.cc:492] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055: Bootstrap starting.
I20260927 04:06:57.514541 5837 tablet_bootstrap.cc:654] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:57.514982 5837 log.cc:824] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:57.515014 5838 tablet_bootstrap.cc:654] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:57.515081 5836 tablet_bootstrap.cc:492] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Bootstrap starting.
I20260927 04:06:57.515332 5838 log.cc:824] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:57.515736 5836 tablet_bootstrap.cc:654] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:57.515797 5837 tablet_bootstrap.cc:492] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: No bootstrap required, opened a new log
I20260927 04:06:57.515864 5837 ts_tablet_manager.cc:1397] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 04:06:57.516036 5836 log.cc:824] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:57.516177 5838 tablet_bootstrap.cc:492] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055: No bootstrap required, opened a new log
I20260927 04:06:57.516249 5838 ts_tablet_manager.cc:1397] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 04:06:57.516782 5836 tablet_bootstrap.cc:492] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: No bootstrap required, opened a new log
I20260927 04:06:57.516851 5836 ts_tablet_manager.cc:1397] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 04:06:57.518669 5837 raft_consensus.cc:348] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.518853 5837 raft_consensus.cc:374] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:57.518916 5837 raft_consensus.cc:729] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28afd56259ab4f98976833dd559fba3e, State: Initialized, Role: FOLLOWER
I20260927 04:06:57.518894 5836 raft_consensus.cc:348] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.518975 5838 raft_consensus.cc:348] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.519084 5836 raft_consensus.cc:374] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:57.519065 5837 consensus_queue.cc:260] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.519119 5836 raft_consensus.cc:729] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3bdfc14dcc44c989afea441acfca302, State: Initialized, Role: FOLLOWER
I20260927 04:06:57.519122 5838 raft_consensus.cc:374] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:57.519160 5838 raft_consensus.cc:729] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019243d5ca2e4325b1dcf474d32b3055, State: Initialized, Role: FOLLOWER
I20260927 04:06:57.519250 5836 consensus_queue.cc:260] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.519333 5837 ts_tablet_manager.cc:1428] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260927 04:06:57.519291 5838 consensus_queue.cc:260] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.519467 5822 heartbeater.cc:503] Master 127.17.184.254:44817 was elected leader, sending a full tablet report...
I20260927 04:06:57.519531 5836 ts_tablet_manager.cc:1428] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260927 04:06:57.519538 5838 ts_tablet_manager.cc:1428] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 04:06:57.519682 5560 heartbeater.cc:503] Master 127.17.184.254:44817 was elected leader, sending a full tablet report...
I20260927 04:06:57.519735 5691 heartbeater.cc:503] Master 127.17.184.254:44817 was elected leader, sending a full tablet report...
W20260927 04:06:57.596586 5692 tablet.cc:2369] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 04:06:57.607494 5844 raft_consensus.cc:482] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 04:06:57.607617 5844 raft_consensus.cc:504] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.608086 5844 leader_election.cc:288] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3bdfc14dcc44c989afea441acfca302 (127.17.184.194:43855), 28afd56259ab4f98976833dd559fba3e (127.17.184.195:44249)
I20260927 04:06:57.610962 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.608258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.195:44249 (local address 127.17.184.193:47611)
0927 04:06:57.608329 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.608342 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:57.608388 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:57.608551 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:57.608552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:57.608553 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:57.608630 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.608631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.609348 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.609348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.610116 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.610118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.610676 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.610676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.610752 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.610755 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 04:06:57.610817 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 04:06:57.610859 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 04:06:57.610962 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.608370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:47611 (local address 127.17.184.195:44249)
0927 04:06:57.608417 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.608419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.608428 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:57.608458 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.608459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.608467 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.608548 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:57.608712 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.609259 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.610219 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.610584 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.610677 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.610686 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.610835 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.610877 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 04:06:57.611109 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.608354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.184.193:39471 (local address 127.17.184.194:43855)
0927 04:06:57.608415 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.608417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.608534 (+ 117us) server_negotiation.cc:407] Connection header received
0927 04:06:57.608604 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.608605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.608617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.608712 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 04:06:57.608874 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.609339 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.610277 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.610703 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.610784 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.610792 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.611003 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.611047 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:06:57.611119 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.608279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.194:43855 (local address 127.17.184.193:39471)
0927 04:06:57.608469 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.608477 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:06:57.608550 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:06:57.608727 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:06:57.608728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:06:57.608730 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 04:06:57.608803 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.608805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.609427 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.609428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.610181 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:06:57.610183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.610797 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.610798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:06:57.610875 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.610877 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 04:06:57.610994 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 04:06:57.611034 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":108,"threads_started":1}
I20260927 04:06:57.611327 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd906752b94c9095732c89a7282643" candidate_uuid: "019243d5ca2e4325b1dcf474d32b3055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28afd56259ab4f98976833dd559fba3e" is_pre_election: true
I20260927 04:06:57.611383 5645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd906752b94c9095732c89a7282643" candidate_uuid: "019243d5ca2e4325b1dcf474d32b3055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3bdfc14dcc44c989afea441acfca302" is_pre_election: true
I20260927 04:06:57.611511 5776 raft_consensus.cc:2454] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019243d5ca2e4325b1dcf474d32b3055 in term 0.
I20260927 04:06:57.611526 5645 raft_consensus.cc:2454] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019243d5ca2e4325b1dcf474d32b3055 in term 0.
I20260927 04:06:57.611769 5447 leader_election.cc:302] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019243d5ca2e4325b1dcf474d32b3055, c3bdfc14dcc44c989afea441acfca302; no voters:
I20260927 04:06:57.611943 5844 raft_consensus.cc:2781] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 04:06:57.612004 5844 raft_consensus.cc:482] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 04:06:57.612052 5844 raft_consensus.cc:3037] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:57.612761 5844 raft_consensus.cc:504] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.612941 5844 leader_election.cc:288] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [CANDIDATE]: Term 1 election: Requested vote from peers c3bdfc14dcc44c989afea441acfca302 (127.17.184.194:43855), 28afd56259ab4f98976833dd559fba3e (127.17.184.195:44249)
I20260927 04:06:57.613047 5645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd906752b94c9095732c89a7282643" candidate_uuid: "019243d5ca2e4325b1dcf474d32b3055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3bdfc14dcc44c989afea441acfca302"
I20260927 04:06:57.613130 5776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fd906752b94c9095732c89a7282643" candidate_uuid: "019243d5ca2e4325b1dcf474d32b3055" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28afd56259ab4f98976833dd559fba3e"
I20260927 04:06:57.613144 5645 raft_consensus.cc:3037] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:57.613195 5776 raft_consensus.cc:3037] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:57.614037 5645 raft_consensus.cc:2454] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019243d5ca2e4325b1dcf474d32b3055 in term 1.
I20260927 04:06:57.614037 5776 raft_consensus.cc:2454] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019243d5ca2e4325b1dcf474d32b3055 in term 1.
I20260927 04:06:57.614228 5447 leader_election.cc:302] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019243d5ca2e4325b1dcf474d32b3055, c3bdfc14dcc44c989afea441acfca302; no voters:
I20260927 04:06:57.614346 5844 raft_consensus.cc:2781] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:57.614573 5844 raft_consensus.cc:686] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 1 LEADER]: Becoming Leader. State: Replica: 019243d5ca2e4325b1dcf474d32b3055, State: Running, Role: LEADER
I20260927 04:06:57.614673 5844 consensus_queue.cc:237] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } }
I20260927 04:06:57.615626 5371 catalog_manager.cc:5495] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 reported cstate change: term changed from 0 to 1, leader changed from <none> to 019243d5ca2e4325b1dcf474d32b3055 (127.17.184.193). New cstate: current_term: 1 leader_uuid: "019243d5ca2e4325b1dcf474d32b3055" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "019243d5ca2e4325b1dcf474d32b3055" member_type: VOTER last_known_addr { host: "127.17.184.193" port: 44185 } health_report { overall_health: HEALTHY } } }
I20260927 04:06:57.628852 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.626375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38354 (local address 127.17.184.193:44185)
0927 04:06:57.626437 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:57.626440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:57.626449 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:06:57.626493 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:57.626494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:57.626512 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:57.626604 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:06:57.626734 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.627276 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.628036 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:57.628242 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:57.628322 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:57.628488 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:06:57.628601 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:06:57.628698 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:57.628722 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 04:06:57.628741 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 04:06:57.630836 5776 raft_consensus.cc:1261] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 1 FOLLOWER]: Refusing update from remote peer 019243d5ca2e4325b1dcf474d32b3055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:57.630836 5645 raft_consensus.cc:1261] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 1 FOLLOWER]: Refusing update from remote peer 019243d5ca2e4325b1dcf474d32b3055: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:06:57.631201 5844 consensus_queue.cc:1035] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28afd56259ab4f98976833dd559fba3e" member_type: VOTER last_known_addr { host: "127.17.184.195" port: 44249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:57.631356 5846 consensus_queue.cc:1035] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3bdfc14dcc44c989afea441acfca302" member_type: VOTER last_known_addr { host: "127.17.184.194" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:06:57.633531 5849 mvcc.cc:205] Tried to move back new op lower bound from 7333814344212979712 to 7333814344150536192. Current Snapshot: MvccSnapshot[applied={T|T < 7333814344212979712}]
I20260927 04:06:57.633750 5850 mvcc.cc:205] Tried to move back new op lower bound from 7333814344212979712 to 7333814344150536192. Current Snapshot: MvccSnapshot[applied={T|T < 7333814344212979712}]
I20260927 04:06:57.635831 5370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:57.635926 5370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 04:06:57.636770 5853 mvcc.cc:205] Tried to move back new op lower bound from 7333814344212979712 to 7333814344150536192. Current Snapshot: MvccSnapshot[applied={T|T < 7333814344212979712 or (T in {7333814344212979712})}]
I20260927 04:06:57.636840 5370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e8f55b08c4e74246a0fb1d5f13134af9: Sending DeleteTablet for 3 replicas of tablet a9fd906752b94c9095732c89a7282643
I20260927 04:06:57.637198 5494 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fd906752b94c9095732c89a7282643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:57 UTC) from {username='slave'} at 127.0.0.1:38352
I20260927 04:06:57.637212 5756 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fd906752b94c9095732c89a7282643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:57 UTC) from {username='slave'} at 127.0.0.1:43090
I20260927 04:06:57.637398 5625 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fd906752b94c9095732c89a7282643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 04:06:57 UTC) from {username='slave'} at 127.0.0.1:50862
I20260927 04:06:57.637431 5859 tablet_replica.cc:330] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: stopping tablet replica
I20260927 04:06:57.637523 5859 raft_consensus.cc:2229] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:57.637589 5861 tablet_replica.cc:330] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: stopping tablet replica
I20260927 04:06:57.637615 5859 raft_consensus.cc:2258] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:57.637777 5860 tablet_replica.cc:330] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055: stopping tablet replica
I20260927 04:06:57.637861 5860 raft_consensus.cc:2229] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 1 LEADER]: Raft consensus shutting down.
I20260927 04:06:57.637939 5860 raft_consensus.cc:2258] T a9fd906752b94c9095732c89a7282643 P 019243d5ca2e4325b1dcf474d32b3055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:57.638042 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 5432
I20260927 04:06:57.638072 5859 ts_tablet_manager.cc:1905] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:57.640051 5859 ts_tablet_manager.cc:1918] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:57.640125 5859 log.cc:1196] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-2/wal/wals/a9fd906752b94c9095732c89a7282643
I20260927 04:06:57.640367 5859 ts_tablet_manager.cc:1939] T a9fd906752b94c9095732c89a7282643 P 28afd56259ab4f98976833dd559fba3e: Deleting consensus metadata
I20260927 04:06:57.637683 5861 raft_consensus.cc:2229] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 04:06:57.640635 5861 raft_consensus.cc:2258] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 04:06:57.640906 5357 catalog_manager.cc:4841] TS 28afd56259ab4f98976833dd559fba3e (127.17.184.195:44249): tablet a9fd906752b94c9095732c89a7282643 (table test-table [id=ac8e04f7935246699b03be659a4aeea3]) successfully deleted
I20260927 04:06:57.641088 5861 ts_tablet_manager.cc:1905] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 04:06:57.642956 5861 ts_tablet_manager.cc:1918] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 04:06:57.643024 5861 log.cc:1196] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Deleting WAL directory at /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790481773036938-18147-0/minicluster-data/ts-1/wal/wals/a9fd906752b94c9095732c89a7282643
I20260927 04:06:57.643253 5861 ts_tablet_manager.cc:1939] T a9fd906752b94c9095732c89a7282643 P c3bdfc14dcc44c989afea441acfca302: Deleting consensus metadata
I20260927 04:06:57.643669 5356 catalog_manager.cc:4841] TS c3bdfc14dcc44c989afea441acfca302 (127.17.184.194:43855): tablet a9fd906752b94c9095732c89a7282643 (table test-table [id=ac8e04f7935246699b03be659a4aeea3]) successfully deleted
W20260927 04:06:57.644944 5357 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.184.193:44185 (error 108)
I20260927 04:06:57.645296 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 5563
I20260927 04:06:57.645388 5830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:57.645249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.193:44185 (local address 127.0.0.1:38362)
0927 04:06:57.645306 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 04:06:57.645350 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.193:44185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 04:06:57.645963 5357 catalog_manager.cc:4584] TS 019243d5ca2e4325b1dcf474d32b3055 (127.17.184.193:44185): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9fd906752b94c9095732c89a7282643: Network error: Client connection negotiation failed: client connection to 127.17.184.193:44185: connect: Connection refused (error 111)
I20260927 04:06:57.650571 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 5694
I20260927 04:06:57.655383 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 5341
2026-09-27T04:06:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (769 ms)
[----------] 6 tests from EncryptionPolicyTest (8709 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:06:57 dist-test-slave-zqrn krb5kdc[5865](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:06:57 dist-test-slave-zqrn krb5kdc[5865](info): set up 2 sockets
Sep 27 04:06:57 dist-test-slave-zqrn krb5kdc[5865](info): commencing operation
krb5kdc: starting...
W20260927 04:06:58.707026 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:06:58 dist-test-slave-zqrn krb5kdc[5865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:06:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:06:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:06:58.827848 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:06:58.912070 5878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:06:58.912293 5878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:06:58.912317 5878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:06:58.913439 5878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:06:58.913475 5878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:06:58.913488 5878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:06:58.913501 5878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:06:58.913512 5878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:06:58.914911 5878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:06:58.915166 5878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:06:58.915331 5878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:06:58.920183 5886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:58.920261 5883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:06:58.920235 5884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:06:58.920363 5878 server_base.cc:1034] running on GCE node
I20260927 04:06:58.920643 5878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:06:58.920982 5878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:06:58.922183 5878 hybrid_clock.cc:648] HybridClock initialized: now 1790482018922154 us; error 40 us; skew 500 ppm
Sep 27 04:06:58 dist-test-slave-zqrn krb5kdc[5865](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:06:58.924878 5878 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:06:58.925364 5878 webserver.cc:460] Webserver started at http://127.0.0.1:34181/ using document root <none> and password file <none>
I20260927 04:06:58.925592 5878 fs_manager.cc:359] Metadata directory not provided
I20260927 04:06:58.925660 5878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:06:58.925812 5878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:06:58.926867 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "e4e3d54d1d8240358c6dfea2792cae3b"
format_stamp: "Formatted at 2026-09-27 04:06:58 on dist-test-slave-zqrn"
server_key: "399b6fc16dd03e56d10fda9f811b62e2"
server_key_iv: "a0ee98984dee14f9e51ca301b9afaa76"
server_key_version: "encryptionkey@0"
I20260927 04:06:58.927312 5878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "e4e3d54d1d8240358c6dfea2792cae3b"
format_stamp: "Formatted at 2026-09-27 04:06:58 on dist-test-slave-zqrn"
server_key: "399b6fc16dd03e56d10fda9f811b62e2"
server_key_iv: "a0ee98984dee14f9e51ca301b9afaa76"
server_key_version: "encryptionkey@0"
I20260927 04:06:58.928669 5878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:06:58.929414 5893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:06:58.929589 5878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:06:58.929689 5878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "e4e3d54d1d8240358c6dfea2792cae3b"
format_stamp: "Formatted at 2026-09-27 04:06:58 on dist-test-slave-zqrn"
server_key: "399b6fc16dd03e56d10fda9f811b62e2"
server_key_iv: "a0ee98984dee14f9e51ca301b9afaa76"
server_key_version: "encryptionkey@0"
I20260927 04:06:58.929791 5878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:06:58.952250 5878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:06:58.954866 5878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:06:58.955071 5878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:06:58.959108 5878 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41533
I20260927 04:06:58.959142 5945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41533 every 8 connection(s)
I20260927 04:06:58.959529 5878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:06:58.960394 5946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:06:58.962687 5946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b: Bootstrap starting.
I20260927 04:06:58.963068 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 5878
I20260927 04:06:58.963191 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:06:58.963372 18147 external_mini_cluster.cc:1428] Setting key 13b145eb47fa147cfb25f0b5ab3148c8
I20260927 04:06:58.963467 5946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b: Neither blocks nor log segments found. Creating new log.
I20260927 04:06:58.963800 5946 log.cc:824] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b: Log is configured to *not* fsync() on all Append() calls
I20260927 04:06:58.964553 5946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b: No bootstrap required, opened a new log
I20260927 04:06:58.966535 5946 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3d54d1d8240358c6dfea2792cae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260927 04:06:58.966670 5946 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:06:58.966699 5946 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e3d54d1d8240358c6dfea2792cae3b, State: Initialized, Role: FOLLOWER
I20260927 04:06:58.966854 5946 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3d54d1d8240358c6dfea2792cae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260927 04:06:58.966958 5946 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:06:58.967000 5946 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:06:58.967047 5946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:06:58.967734 5946 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3d54d1d8240358c6dfea2792cae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260927 04:06:58.967859 5946 leader_election.cc:302] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4e3d54d1d8240358c6dfea2792cae3b; no voters:
I20260927 04:06:58.968045 5946 leader_election.cc:288] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:06:58.968071 5951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:06:58.968192 5951 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [term 1 LEADER]: Becoming Leader. State: Replica: e4e3d54d1d8240358c6dfea2792cae3b, State: Running, Role: LEADER
I20260927 04:06:58.968276 5951 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3d54d1d8240358c6dfea2792cae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } }
I20260927 04:06:58.968314 5946 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:06:58.968600 5952 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4e3d54d1d8240358c6dfea2792cae3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3d54d1d8240358c6dfea2792cae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } } }
I20260927 04:06:58.968873 5952 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:58.968796 5953 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4e3d54d1d8240358c6dfea2792cae3b. Latest consensus state: current_term: 1 leader_uuid: "e4e3d54d1d8240358c6dfea2792cae3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4e3d54d1d8240358c6dfea2792cae3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41533 } } }
I20260927 04:06:58.969115 5953 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b [sys.catalog]: This master's current role is: LEADER
I20260927 04:06:58.969219 5960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:06:58.970564 5960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 04:06:58 dist-test-slave-zqrn krb5kdc[5865](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 04:06:58.972719 5960 catalog_manager.cc:1347] Generated new cluster ID: 502c0a235cb040b29aaf851ffc9c1734
I20260927 04:06:58.972772 5960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:06:58.975435 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:58.964088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37677 (local address 127.0.0.1:41533)
0927 04:06:58.964267 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:58.964277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:58.964364 (+ 87us) server_negotiation.cc:407] Connection header received
0927 04:06:58.964913 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:58.964920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:58.964970 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:58.965116 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:58.965796 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:58.966639 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:58.967826 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:58.968500 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:58.968922 (+ 422us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:58.971603 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:58.971607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:58.971608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:58.971627 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:58.973650 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:58.974124 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:58.974127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:58.974127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:58.974239 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:58.974473 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:58.974474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:58.974474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:58.974800 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:58.974986 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:58.975104 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 04:06:58.975212 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260927 04:06:58.976420 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:06:58.985265 5950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:06:58.977964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50739 (local address 127.0.0.1:41533)
0927 04:06:58.978026 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0927 04:06:58.978028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:06:58.978214 (+ 186us) server_negotiation.cc:407] Connection header received
0927 04:06:58.978275 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:06:58.978276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:06:58.978285 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:06:58.978338 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:06:58.979001 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:58.979733 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:58.980750 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:06:58.981053 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:06:58.981110 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:06:58.982521 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:06:58.982522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:06:58.982522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:06:58.982529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:06:58.984049 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:58.984429 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:58.984429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:58.984430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:58.984478 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:06:58.984699 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:06:58.984699 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:06:58.984700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:06:58.984817 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:06:58.984889 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:06:58.985110 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 04:06:58.985170 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:06:58.993575 5960 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:06:58.994110 5960 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:06:59.003188 5960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4e3d54d1d8240358c6dfea2792cae3b: Generated new TSK 0
I20260927 04:06:59.003412 5960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:06:59.005681 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 5878
W20260927 04:06:59.009720 5863 connection.cc:535] client connection to 127.0.0.1:41533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41533): Connection reset by peer (error 104)
2026-09-27T04:06:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:06:59 dist-test-slave-zqrn krb5kdc[5977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:06:59 dist-test-slave-zqrn krb5kdc[5977](info): set up 2 sockets
Sep 27 04:06:59 dist-test-slave-zqrn krb5kdc[5977](info): commencing operation
krb5kdc: starting...
W20260927 04:07:00.079030 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:00 dist-test-slave-zqrn krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:00.199747 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:00.281255 5990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:00.281514 5990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:00.281551 5990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:00.282714 5990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:00.282790 5990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:00.282816 5990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:00.282846 5990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:00.282888 5990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:00.284375 5990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:00.284655 5990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:00.284883 5990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:00.289880 5998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:00.289911 5990 server_base.cc:1034] running on GCE node
W20260927 04:07:00.289903 5995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:00.289880 5996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:00.290423 5990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:00.290726 5990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:00.291891 5990 hybrid_clock.cc:648] HybridClock initialized: now 1790482020291855 us; error 53 us; skew 500 ppm
Sep 27 04:07:00 dist-test-slave-zqrn krb5kdc[5977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:00.294754 5990 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:00.295307 5990 webserver.cc:460] Webserver started at http://127.0.0.1:44457/ using document root <none> and password file <none>
I20260927 04:07:00.295558 5990 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:00.295608 5990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:00.295749 5990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:00.296901 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "f951a85792e34557ae7b51ada4bb76e3"
format_stamp: "Formatted at 2026-09-27 04:07:00 on dist-test-slave-zqrn"
server_key: "80a6e6e9a7dc595d644da1d6372cbf42"
server_key_iv: "1fc07fa1450c654901152cc7b053ac81"
server_key_version: "encryptionkey@0"
I20260927 04:07:00.297298 5990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "f951a85792e34557ae7b51ada4bb76e3"
format_stamp: "Formatted at 2026-09-27 04:07:00 on dist-test-slave-zqrn"
server_key: "80a6e6e9a7dc595d644da1d6372cbf42"
server_key_iv: "1fc07fa1450c654901152cc7b053ac81"
server_key_version: "encryptionkey@0"
I20260927 04:07:00.298823 5990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:07:00.299736 6005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:00.299939 5990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:00.300055 5990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "f951a85792e34557ae7b51ada4bb76e3"
format_stamp: "Formatted at 2026-09-27 04:07:00 on dist-test-slave-zqrn"
server_key: "80a6e6e9a7dc595d644da1d6372cbf42"
server_key_iv: "1fc07fa1450c654901152cc7b053ac81"
server_key_version: "encryptionkey@0"
I20260927 04:07:00.300153 5990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:00.312330 5990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:00.314651 5990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:00.314829 5990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:00.319271 5990 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33421
I20260927 04:07:00.319304 6057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33421 every 8 connection(s)
I20260927 04:07:00.319725 5990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:00.320760 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:00.323369 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3: Bootstrap starting.
I20260927 04:07:00.324178 6058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:00.324404 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 5990
I20260927 04:07:00.324509 6058 log.cc:824] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:00.324535 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:00.324848 18147 external_mini_cluster.cc:1428] Setting key aa8cccc38df673774e678bfc1d069568
I20260927 04:07:00.325345 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3: No bootstrap required, opened a new log
I20260927 04:07:00.327178 6058 raft_consensus.cc:348] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f951a85792e34557ae7b51ada4bb76e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33421 } }
I20260927 04:07:00.327342 6058 raft_consensus.cc:374] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:00.327389 6058 raft_consensus.cc:729] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f951a85792e34557ae7b51ada4bb76e3, State: Initialized, Role: FOLLOWER
I20260927 04:07:00.327521 6058 consensus_queue.cc:260] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f951a85792e34557ae7b51ada4bb76e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33421 } }
I20260927 04:07:00.327623 6058 raft_consensus.cc:388] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:00.327670 6058 raft_consensus.cc:482] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:00.327728 6058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:00.328317 6058 raft_consensus.cc:504] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f951a85792e34557ae7b51ada4bb76e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33421 } }
I20260927 04:07:00.328405 6058 leader_election.cc:302] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f951a85792e34557ae7b51ada4bb76e3; no voters:
I20260927 04:07:00.328593 6058 leader_election.cc:288] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:00.328673 6063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:00.328836 6063 raft_consensus.cc:686] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [term 1 LEADER]: Becoming Leader. State: Replica: f951a85792e34557ae7b51ada4bb76e3, State: Running, Role: LEADER
I20260927 04:07:00.328852 6058 sys_catalog.cc:564] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:00.328936 6063 consensus_queue.cc:237] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f951a85792e34557ae7b51ada4bb76e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33421 } }
I20260927 04:07:00.329264 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f951a85792e34557ae7b51ada4bb76e3. Latest consensus state: current_term: 1 leader_uuid: "f951a85792e34557ae7b51ada4bb76e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f951a85792e34557ae7b51ada4bb76e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33421 } } }
I20260927 04:07:00.329331 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:00.329299 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f951a85792e34557ae7b51ada4bb76e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f951a85792e34557ae7b51ada4bb76e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33421 } } }
I20260927 04:07:00.329587 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:00.329649 6071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:00.330210 6071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:00.331885 6071 catalog_manager.cc:1347] Generated new cluster ID: c60a31d53d53478aac1fd17846ded89e
I20260927 04:07:00.331931 6071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:00.333029 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:00.325353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40221 (local address 127.0.0.1:33421)
0927 04:07:00.325558 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:00.325569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:00.325600 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:07:00.327164 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:00.327172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:00.327231 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:00.327389 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:00.327648 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:00.329398 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:00.330746 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:00.331298 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:00.331440 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:00.332308 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:00.332312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:00.332313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:00.332534 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:00.332570 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:00.332582 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:00.332686 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:00.332775 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 04:07:00.332804 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260927 04:07:00.334021 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:00.338665 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:00.335523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44421 (local address 127.0.0.1:33421)
0927 04:07:00.335561 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:00.335563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:00.335692 (+ 129us) server_negotiation.cc:407] Connection header received
0927 04:07:00.335770 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:00.335771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:00.335782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:00.335844 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:00.336013 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:00.336517 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:00.337299 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:00.337560 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:00.337639 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:00.338385 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:00.338387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:00.338387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:00.338392 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:00.338406 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:00.338411 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:00.338479 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:00.338546 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 04:07:00.338558 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:07:00.346377 6071 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:00.347210 6071 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:00.352427 6071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f951a85792e34557ae7b51ada4bb76e3: Generated new TSK 0
I20260927 04:07:00.352633 6071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:00.355860 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 5990
W20260927 04:07:00.360023 5975 connection.cc:535] client connection to 127.0.0.1:33421 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33421): Connection reset by peer (error 104)
2026-09-27T04:07:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:00 dist-test-slave-zqrn krb5kdc[6089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:00 dist-test-slave-zqrn krb5kdc[6089](info): set up 2 sockets
Sep 27 04:07:00 dist-test-slave-zqrn krb5kdc[6089](info): commencing operation
krb5kdc: starting...
W20260927 04:07:01.395040 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:01 dist-test-slave-zqrn krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:01.522995 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:01.605593 6102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:01.605852 6102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:01.605893 6102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:01.607136 6102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:01.607198 6102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:01.607223 6102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:01.607255 6102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:01.607280 6102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:01.608803 6102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:01.609092 6102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:01.609328 6102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:01.614660 6110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:01.614635 6108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:01.614667 6102 server_base.cc:1034] running on GCE node
W20260927 04:07:01.614631 6107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:01.615129 6102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:01.615444 6102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:01.616636 6102 hybrid_clock.cc:648] HybridClock initialized: now 1790482021616622 us; error 32 us; skew 500 ppm
Sep 27 04:07:01 dist-test-slave-zqrn krb5kdc[6089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:01.619436 6102 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:01.619951 6102 webserver.cc:460] Webserver started at http://127.0.0.1:44225/ using document root <none> and password file <none>
I20260927 04:07:01.620205 6102 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:01.620276 6102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:01.620425 6102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:01.621533 6102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "04263bd1f9534dee88e9cbafd91a2a63"
format_stamp: "Formatted at 2026-09-27 04:07:01 on dist-test-slave-zqrn"
server_key: "d6dba5ec6eae317d3d2e48d31c5a6ec5"
server_key_iv: "16559ebe4327fec4c5cc50973da2dafc"
server_key_version: "encryptionkey@0"
I20260927 04:07:01.621891 6102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "04263bd1f9534dee88e9cbafd91a2a63"
format_stamp: "Formatted at 2026-09-27 04:07:01 on dist-test-slave-zqrn"
server_key: "d6dba5ec6eae317d3d2e48d31c5a6ec5"
server_key_iv: "16559ebe4327fec4c5cc50973da2dafc"
server_key_version: "encryptionkey@0"
I20260927 04:07:01.623306 6102 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:07:01.624094 6117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:01.624307 6102 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:07:01.624418 6102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "04263bd1f9534dee88e9cbafd91a2a63"
format_stamp: "Formatted at 2026-09-27 04:07:01 on dist-test-slave-zqrn"
server_key: "d6dba5ec6eae317d3d2e48d31c5a6ec5"
server_key_iv: "16559ebe4327fec4c5cc50973da2dafc"
server_key_version: "encryptionkey@0"
I20260927 04:07:01.624491 6102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:01.627900 6102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:01.630223 6102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:01.630379 6102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:01.634420 6102 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34561
I20260927 04:07:01.634452 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34561 every 8 connection(s)
I20260927 04:07:01.634812 6102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:01.635663 6170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:01.637887 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6102
I20260927 04:07:01.637887 6170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63: Bootstrap starting.
I20260927 04:07:01.638003 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:01.638190 18147 external_mini_cluster.cc:1428] Setting key fcf18fc644841b57170462f9367044ef
I20260927 04:07:01.638585 6170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:01.638970 6170 log.cc:824] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:01.639689 6170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63: No bootstrap required, opened a new log
I20260927 04:07:01.640694 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:01.638710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41795 (local address 127.0.0.1:34561)
0927 04:07:01.638931 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:01.638939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:01.638985 (+ 46us) server_negotiation.cc:407] Connection header received
0927 04:07:01.639469 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:01.639475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:01.639507 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:01.639629 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:01.640253 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:01.640256 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:01.640257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:01.640260 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:01.640294 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:01.640305 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:01.640377 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:01.640440 (+ 63us) server_negotiation.cc:299] Negotiation successful
0927 04:07:01.640495 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260927 04:07:01.641621 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:01.642096 6170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04263bd1f9534dee88e9cbafd91a2a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260927 04:07:01.642225 6170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:01.642246 6170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04263bd1f9534dee88e9cbafd91a2a63, State: Initialized, Role: FOLLOWER
I20260927 04:07:01.642366 6170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04263bd1f9534dee88e9cbafd91a2a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260927 04:07:01.642448 6170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:01.642489 6170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:01.642537 6170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:01.643172 6170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04263bd1f9534dee88e9cbafd91a2a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260927 04:07:01.643286 6170 leader_election.cc:302] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04263bd1f9534dee88e9cbafd91a2a63; no voters:
I20260927 04:07:01.643448 6170 leader_election.cc:288] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:01.643471 6180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:01.643579 6180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [term 1 LEADER]: Becoming Leader. State: Replica: 04263bd1f9534dee88e9cbafd91a2a63, State: Running, Role: LEADER
I20260927 04:07:01.643709 6170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:01.644174 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:01.643117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35367 (local address 127.0.0.1:34561)
0927 04:07:01.643157 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:01.643159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:01.643281 (+ 122us) server_negotiation.cc:407] Connection header received
0927 04:07:01.643367 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:01.643368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:01.643373 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:01.643433 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:01.643977 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:01.643978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:01.643978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:01.643983 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:01.643995 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:01.644000 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:01.644058 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:01.644101 (+ 43us) server_negotiation.cc:299] Negotiation successful
0927 04:07:01.644113 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:07:01.643704 6180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04263bd1f9534dee88e9cbafd91a2a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } }
I20260927 04:07:01.645493 6181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04263bd1f9534dee88e9cbafd91a2a63. Latest consensus state: current_term: 1 leader_uuid: "04263bd1f9534dee88e9cbafd91a2a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04263bd1f9534dee88e9cbafd91a2a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } } }
I20260927 04:07:01.645576 6182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04263bd1f9534dee88e9cbafd91a2a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04263bd1f9534dee88e9cbafd91a2a63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34561 } } }
I20260927 04:07:01.645839 6182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:01.645836 6181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63 [sys.catalog]: This master's current role is: LEADER
W20260927 04:07:01.645967 6194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:01.646075 6194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:01.646129 6197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:01.646271 6197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:01.647812 6197 catalog_manager.cc:1347] Generated new cluster ID: 57b44ae8b56e4095aef71c76a1713b14
I20260927 04:07:01.647861 6197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:01.662813 6197 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:01.663025 6197 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:01.669421 6197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04263bd1f9534dee88e9cbafd91a2a63: Generated new TSK 0
I20260927 04:07:01.669613 6197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:01.671584 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6102
2026-09-27T04:07:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:01 dist-test-slave-zqrn krb5kdc[6201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:01 dist-test-slave-zqrn krb5kdc[6201](info): set up 2 sockets
Sep 27 04:07:01 dist-test-slave-zqrn krb5kdc[6201](info): commencing operation
krb5kdc: starting...
W20260927 04:07:02.719090 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:02 dist-test-slave-zqrn krb5kdc[6201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:02.841481 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:02.921797 6214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:02.921993 6214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:02.922014 6214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:02.923187 6214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:02.923228 6214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:02.923241 6214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:02.923254 6214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:02.923265 6214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:02.924746 6214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:02.925002 6214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:02.925256 6214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:02.930140 6219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:02.930182 6220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:02.930234 6222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:02.930321 6214 server_base.cc:1034] running on GCE node
I20260927 04:07:02.930650 6214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:02.930935 6214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:02.932122 6214 hybrid_clock.cc:648] HybridClock initialized: now 1790482022932101 us; error 41 us; skew 500 ppm
Sep 27 04:07:02 dist-test-slave-zqrn krb5kdc[6201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482022, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:02.934603 6214 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:02.935132 6214 webserver.cc:460] Webserver started at http://127.0.0.1:45667/ using document root <none> and password file <none>
I20260927 04:07:02.935376 6214 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:02.935425 6214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:02.935551 6214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:02.936651 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "a2fd07e42bda4d07aeb20a3786ed656f"
format_stamp: "Formatted at 2026-09-27 04:07:02 on dist-test-slave-zqrn"
server_key: "1691cfdff24e4f2c7c55c569d87e92df"
server_key_iv: "66f0c5c1e6da846d7b072c771144a720"
server_key_version: "encryptionkey@0"
I20260927 04:07:02.936998 6214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "a2fd07e42bda4d07aeb20a3786ed656f"
format_stamp: "Formatted at 2026-09-27 04:07:02 on dist-test-slave-zqrn"
server_key: "1691cfdff24e4f2c7c55c569d87e92df"
server_key_iv: "66f0c5c1e6da846d7b072c771144a720"
server_key_version: "encryptionkey@0"
I20260927 04:07:02.938364 6214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:07:02.939213 6229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:02.939424 6214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:02.939533 6214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "a2fd07e42bda4d07aeb20a3786ed656f"
format_stamp: "Formatted at 2026-09-27 04:07:02 on dist-test-slave-zqrn"
server_key: "1691cfdff24e4f2c7c55c569d87e92df"
server_key_iv: "66f0c5c1e6da846d7b072c771144a720"
server_key_version: "encryptionkey@0"
I20260927 04:07:02.939596 6214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:02.964063 6214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:02.966598 6214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:02.966768 6214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:02.970700 6214 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37519
I20260927 04:07:02.970744 6281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37519 every 8 connection(s)
I20260927 04:07:02.971154 6214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:02.972105 6282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:02.974794 6282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f: Bootstrap starting.
I20260927 04:07:02.975492 6282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:02.975816 6282 log.cc:824] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:02.976686 6282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f: No bootstrap required, opened a new log
I20260927 04:07:02.976861 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6214
I20260927 04:07:02.976975 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:02.977197 18147 external_mini_cluster.cc:1428] Setting key 3cbbe5f5d8646506567fef43f254b8f5
I20260927 04:07:02.979034 6282 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fd07e42bda4d07aeb20a3786ed656f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } }
I20260927 04:07:02.979216 6282 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:02.979259 6282 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2fd07e42bda4d07aeb20a3786ed656f, State: Initialized, Role: FOLLOWER
I20260927 04:07:02.979440 6282 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fd07e42bda4d07aeb20a3786ed656f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } }
I20260927 04:07:02.979545 6282 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:02.979581 6282 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:02.979640 6282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:02.980721 6282 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fd07e42bda4d07aeb20a3786ed656f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } }
I20260927 04:07:02.980815 6282 leader_election.cc:302] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2fd07e42bda4d07aeb20a3786ed656f; no voters:
I20260927 04:07:02.981027 6282 leader_election.cc:288] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:02.981074 6287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:02.981240 6287 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [term 1 LEADER]: Becoming Leader. State: Replica: a2fd07e42bda4d07aeb20a3786ed656f, State: Running, Role: LEADER
I20260927 04:07:02.981320 6282 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:02.981388 6287 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fd07e42bda4d07aeb20a3786ed656f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } }
I20260927 04:07:02.981739 6287 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2fd07e42bda4d07aeb20a3786ed656f. Latest consensus state: current_term: 1 leader_uuid: "a2fd07e42bda4d07aeb20a3786ed656f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fd07e42bda4d07aeb20a3786ed656f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } } }
I20260927 04:07:02.981757 6288 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2fd07e42bda4d07aeb20a3786ed656f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2fd07e42bda4d07aeb20a3786ed656f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37519 } } }
I20260927 04:07:02.981933 6288 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:02.982126 6287 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:02.982297 6294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:02.982800 6294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:02.984932 6294 catalog_manager.cc:1347] Generated new cluster ID: 8dc43d23b0a54837b678edbc9d70a82f
I20260927 04:07:02.985002 6294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:07:02 dist-test-slave-zqrn krb5kdc[6201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 04:07:02.990768 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:02.977827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58689 (local address 127.0.0.1:37519)
0927 04:07:02.978008 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:02.978019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:02.978048 (+ 29us) server_negotiation.cc:407] Connection header received
0927 04:07:02.979020 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:02.979028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:02.979093 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:02.979231 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:02.980106 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:02.981193 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:02.982322 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:02.983065 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:02.983499 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:02.986314 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:02.986318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:02.986318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:02.986331 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:02.988627 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:02.989079 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:02.989083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:02.989084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:02.989202 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:02.989390 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:02.989390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:02.989391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:02.989710 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:02.990300 (+ 590us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:02.990374 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 04:07:02.990531 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260927 04:07:02.991616 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:02.996466 6294 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:02.997180 6294 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:03.001756 6286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:02.993123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46297 (local address 127.0.0.1:37519)
0927 04:07:02.993216 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:02.993218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:02.993384 (+ 166us) server_negotiation.cc:407] Connection header received
0927 04:07:02.993445 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:02.993446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:02.993458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:02.993533 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:02.994228 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:02.995107 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:02.996233 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:02.996636 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:02.996726 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:02.998232 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:02.998233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:02.998234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:02.998242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:03.000147 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:03.000586 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:03.000588 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:03.000588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:03.000652 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:03.000859 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:03.000860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:03.000860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:03.001010 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:03.001130 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:03.001536 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 04:07:03.001639 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":52}
I20260927 04:07:03.003430 6294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2fd07e42bda4d07aeb20a3786ed656f: Generated new TSK 0
I20260927 04:07:03.003638 6294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:03.009245 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6214
2026-09-27T04:07:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:03 dist-test-slave-zqrn krb5kdc[6313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:03 dist-test-slave-zqrn krb5kdc[6313](info): set up 2 sockets
Sep 27 04:07:03 dist-test-slave-zqrn krb5kdc[6313](info): commencing operation
krb5kdc: starting...
W20260927 04:07:04.119091 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:04 dist-test-slave-zqrn krb5kdc[6313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:04.237844 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:04.318248 6326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:04.318459 6326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:04.318480 6326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:04.319666 6326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:04.319710 6326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:04.319725 6326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:04.319737 6326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:04.319749 6326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:04.321221 6326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:04.321475 6326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:04.321698 6326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:04.326562 6334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:04.326560 6332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:04.326529 6331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:04.326817 6326 server_base.cc:1034] running on GCE node
I20260927 04:07:04.327085 6326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:04.327368 6326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:04.328502 6326 hybrid_clock.cc:648] HybridClock initialized: now 1790482024328465 us; error 51 us; skew 500 ppm
Sep 27 04:07:04 dist-test-slave-zqrn krb5kdc[6313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:04.335389 6326 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:04.335892 6326 webserver.cc:460] Webserver started at http://127.0.0.1:43457/ using document root <none> and password file <none>
I20260927 04:07:04.336117 6326 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:04.336164 6326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:04.336302 6326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:04.337365 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "4555d92960ad443192a0ebb2dd2e8678"
format_stamp: "Formatted at 2026-09-27 04:07:04 on dist-test-slave-zqrn"
server_key: "1dec5d41f8ace457f91e5c60b7126ef6"
server_key_iv: "e0c8deea7c0c9e044a37627784b31677"
server_key_version: "encryptionkey@0"
I20260927 04:07:04.337736 6326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "4555d92960ad443192a0ebb2dd2e8678"
format_stamp: "Formatted at 2026-09-27 04:07:04 on dist-test-slave-zqrn"
server_key: "1dec5d41f8ace457f91e5c60b7126ef6"
server_key_iv: "e0c8deea7c0c9e044a37627784b31677"
server_key_version: "encryptionkey@0"
I20260927 04:07:04.339133 6326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:07:04.339855 6341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:04.340071 6326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:07:04.340176 6326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "4555d92960ad443192a0ebb2dd2e8678"
format_stamp: "Formatted at 2026-09-27 04:07:04 on dist-test-slave-zqrn"
server_key: "1dec5d41f8ace457f91e5c60b7126ef6"
server_key_iv: "e0c8deea7c0c9e044a37627784b31677"
server_key_version: "encryptionkey@0"
I20260927 04:07:04.340252 6326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:04.359089 6326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:04.361019 6326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:04.361194 6326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:04.365448 6326 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33681
I20260927 04:07:04.365486 6393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33681 every 8 connection(s)
I20260927 04:07:04.365890 6326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:04.366802 6394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:04.369455 6394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678: Bootstrap starting.
I20260927 04:07:04.370055 6394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:04.370342 6394 log.cc:824] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:04.371132 6394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678: No bootstrap required, opened a new log
I20260927 04:07:04.372386 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6326
I20260927 04:07:04.372484 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:04.372690 18147 external_mini_cluster.cc:1428] Setting key 37c6776bd286ce7dd334764a9d3844dc
I20260927 04:07:04.373431 6394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4555d92960ad443192a0ebb2dd2e8678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } }
I20260927 04:07:04.373638 6394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:04.373677 6394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4555d92960ad443192a0ebb2dd2e8678, State: Initialized, Role: FOLLOWER
I20260927 04:07:04.373816 6394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4555d92960ad443192a0ebb2dd2e8678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } }
I20260927 04:07:04.373905 6394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:04.373994 6394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:04.374055 6394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:04.374918 6394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4555d92960ad443192a0ebb2dd2e8678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } }
I20260927 04:07:04.375036 6394 leader_election.cc:302] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4555d92960ad443192a0ebb2dd2e8678; no voters:
I20260927 04:07:04.375193 6394 leader_election.cc:288] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:04.375247 6399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:04.375391 6399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [term 1 LEADER]: Becoming Leader. State: Replica: 4555d92960ad443192a0ebb2dd2e8678, State: Running, Role: LEADER
I20260927 04:07:04.375447 6394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:04.375510 6399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4555d92960ad443192a0ebb2dd2e8678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } }
I20260927 04:07:04.375979 6400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4555d92960ad443192a0ebb2dd2e8678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4555d92960ad443192a0ebb2dd2e8678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } } }
I20260927 04:07:04.376058 6400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:04.376273 6401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4555d92960ad443192a0ebb2dd2e8678. Latest consensus state: current_term: 1 leader_uuid: "4555d92960ad443192a0ebb2dd2e8678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4555d92960ad443192a0ebb2dd2e8678" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33681 } } }
I20260927 04:07:04.376627 6401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:04.376784 6403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:04.377318 6403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:04.379510 6403 catalog_manager.cc:1347] Generated new cluster ID: 7d2c9e3589044d5b99ddfb48a18761a4
I20260927 04:07:04.379597 6403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:04.379729 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:04.373263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43935 (local address 127.0.0.1:33681)
0927 04:07:04.373482 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:04.373492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:04.373528 (+ 36us) server_negotiation.cc:407] Connection header received
0927 04:07:04.373645 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:04.373652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:04.373730 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:04.373881 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:04.374091 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:04.376114 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:04.377376 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:04.377984 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:04.378096 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:04.378838 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:04.378843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:04.378843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:04.379076 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:04.379109 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:04.379121 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:04.379221 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:04.379406 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 04:07:04.379456 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20260927 04:07:04.380621 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:04.390305 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:04.387210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34001 (local address 127.0.0.1:33681)
0927 04:07:04.387254 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:04.387256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:04.387452 (+ 196us) server_negotiation.cc:407] Connection header received
0927 04:07:04.387530 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:04.387531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:04.387543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:04.387623 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:04.387798 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:04.388274 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:04.389006 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:04.389223 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:04.389309 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:04.390018 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:04.390020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:04.390020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:04.390026 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:04.390041 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:04.390046 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:04.390100 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:04.390161 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 04:07:04.390195 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:07:04.404927 6403 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:04.405679 6403 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:04.414716 6403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4555d92960ad443192a0ebb2dd2e8678: Generated new TSK 0
I20260927 04:07:04.414943 6403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:04.423684 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6326
2026-09-27T04:07:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:04 dist-test-slave-zqrn krb5kdc[6425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:04 dist-test-slave-zqrn krb5kdc[6425](info): set up 2 sockets
Sep 27 04:07:04 dist-test-slave-zqrn krb5kdc[6425](info): commencing operation
krb5kdc: starting...
W20260927 04:07:05.455698 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:05 dist-test-slave-zqrn krb5kdc[6425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:05.573988 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:05.654011 6438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:05.654280 6438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:05.654323 6438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:05.655576 6438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:05.655653 6438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:05.655678 6438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:05.655709 6438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:05.655738 6438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:05.657238 6438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:05.657527 6438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:05.657765 6438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:05.662657 6446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:05.662688 6438 server_base.cc:1034] running on GCE node
W20260927 04:07:05.662707 6443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:05.662693 6444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:05.663203 6438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:05.663447 6438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:05.664602 6438 hybrid_clock.cc:648] HybridClock initialized: now 1790482025664589 us; error 30 us; skew 500 ppm
Sep 27 04:07:05 dist-test-slave-zqrn krb5kdc[6425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:05.667173 6438 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:05.667629 6438 webserver.cc:460] Webserver started at http://127.0.0.1:40663/ using document root <none> and password file <none>
I20260927 04:07:05.667860 6438 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:05.667904 6438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:05.668066 6438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:05.669241 6438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "4eac926b70024205b5897095e507d14d"
format_stamp: "Formatted at 2026-09-27 04:07:05 on dist-test-slave-zqrn"
server_key: "8f97b30fe119422387b1ffd8eeacac46"
server_key_iv: "c03dea292983f5d4cb393b278878300a"
server_key_version: "encryptionkey@0"
I20260927 04:07:05.669601 6438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "4eac926b70024205b5897095e507d14d"
format_stamp: "Formatted at 2026-09-27 04:07:05 on dist-test-slave-zqrn"
server_key: "8f97b30fe119422387b1ffd8eeacac46"
server_key_iv: "c03dea292983f5d4cb393b278878300a"
server_key_version: "encryptionkey@0"
I20260927 04:07:05.671047 6438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:07:05.671989 6453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:05.672183 6438 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:07:05.672272 6438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "4eac926b70024205b5897095e507d14d"
format_stamp: "Formatted at 2026-09-27 04:07:05 on dist-test-slave-zqrn"
server_key: "8f97b30fe119422387b1ffd8eeacac46"
server_key_iv: "c03dea292983f5d4cb393b278878300a"
server_key_version: "encryptionkey@0"
I20260927 04:07:05.672325 6438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:05.675653 6438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:05.677744 6438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:05.677851 6438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:05.681826 6438 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43255
I20260927 04:07:05.681876 6505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43255 every 8 connection(s)
I20260927 04:07:05.682227 6438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:05.683192 6506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:05.685413 6506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d: Bootstrap starting.
I20260927 04:07:05.685989 6506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:05.686273 6506 log.cc:824] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:05.686998 6506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d: No bootstrap required, opened a new log
I20260927 04:07:05.688561 6506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac926b70024205b5897095e507d14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } }
I20260927 04:07:05.688719 6506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:05.688766 6506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eac926b70024205b5897095e507d14d, State: Initialized, Role: FOLLOWER
I20260927 04:07:05.688959 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6438
I20260927 04:07:05.688977 6506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac926b70024205b5897095e507d14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } }
I20260927 04:07:05.689073 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:05.689073 6506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:05.689110 6506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:05.689160 6506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:05.689298 18147 external_mini_cluster.cc:1428] Setting key a5bd9925cb336809ad9bd5f2c486866c
I20260927 04:07:05.689826 6506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac926b70024205b5897095e507d14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } }
I20260927 04:07:05.689983 6506 leader_election.cc:302] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4eac926b70024205b5897095e507d14d; no voters:
I20260927 04:07:05.690140 6506 leader_election.cc:288] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:05.690166 6511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:05.690266 6511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [term 1 LEADER]: Becoming Leader. State: Replica: 4eac926b70024205b5897095e507d14d, State: Running, Role: LEADER
I20260927 04:07:05.690392 6506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:05.690409 6511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac926b70024205b5897095e507d14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } }
I20260927 04:07:05.691388 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4eac926b70024205b5897095e507d14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac926b70024205b5897095e507d14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } } }
I20260927 04:07:05.691507 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:05.691569 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:05.689760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46019 (local address 127.0.0.1:43255)
0927 04:07:05.689955 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:05.689964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:05.690002 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:07:05.690118 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:05.690125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:05.690160 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:05.690577 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:05.691010 (+ 433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:05.691013 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:05.691013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:05.691017 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:05.691060 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:05.691075 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:05.691149 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:05.691225 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 04:07:05.691356 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260927 04:07:05.691644 6513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4eac926b70024205b5897095e507d14d. Latest consensus state: current_term: 1 leader_uuid: "4eac926b70024205b5897095e507d14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4eac926b70024205b5897095e507d14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43255 } } }
I20260927 04:07:05.691751 6513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:05.691838 6520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:05.692420 6520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:05.692485 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:05.694432 6520 catalog_manager.cc:1347] Generated new cluster ID: 3cc8867ff68147108daa580885ed16e2
I20260927 04:07:05.694483 6520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:05.695091 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:05.694098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46429 (local address 127.0.0.1:43255)
0927 04:07:05.694133 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:05.694135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:05.694146 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:07:05.694167 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:05.694168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:05.694172 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:05.694241 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:05.694843 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:05.694845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:05.694845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:05.694849 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:05.694866 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:05.694870 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:05.694969 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:05.695017 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 04:07:05.695030 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:07:05.704063 6520 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:05.704250 6520 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:05.712744 6520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4eac926b70024205b5897095e507d14d: Generated new TSK 0
I20260927 04:07:05.712950 6520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:05.716012 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6438
2026-09-27T04:07:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:05 dist-test-slave-zqrn krb5kdc[6537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:05 dist-test-slave-zqrn krb5kdc[6537](info): set up 2 sockets
Sep 27 04:07:05 dist-test-slave-zqrn krb5kdc[6537](info): commencing operation
krb5kdc: starting...
W20260927 04:07:06.763091 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:06 dist-test-slave-zqrn krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:06.884424 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:06.968427 6550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:06.968686 6550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:06.968725 6550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:06.969895 6550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:06.969969 6550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:06.969992 6550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:06.970013 6550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:06.970042 6550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:06.971534 6550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:06.971813 6550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:06.972033 6550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:06.977188 6558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:06.977267 6550 server_base.cc:1034] running on GCE node
W20260927 04:07:06.977200 6555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:06.977198 6556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:06.977648 6550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:06.977988 6550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:06.979158 6550 hybrid_clock.cc:648] HybridClock initialized: now 1790482026979133 us; error 40 us; skew 500 ppm
Sep 27 04:07:06 dist-test-slave-zqrn krb5kdc[6537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:06.981554 6550 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:06.982074 6550 webserver.cc:460] Webserver started at http://127.0.0.1:39219/ using document root <none> and password file <none>
I20260927 04:07:06.982316 6550 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:06.982384 6550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:06.982553 6550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:06.983791 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "1fe2444dc8af45129a6bbaa55fa42051"
format_stamp: "Formatted at 2026-09-27 04:07:06 on dist-test-slave-zqrn"
server_key: "d34bdd7eead8450c32f921eb1d6f865f"
server_key_iv: "687e6f394b949121dcf979ef9f107c37"
server_key_version: "encryptionkey@0"
I20260927 04:07:06.984171 6550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "1fe2444dc8af45129a6bbaa55fa42051"
format_stamp: "Formatted at 2026-09-27 04:07:06 on dist-test-slave-zqrn"
server_key: "d34bdd7eead8450c32f921eb1d6f865f"
server_key_iv: "687e6f394b949121dcf979ef9f107c37"
server_key_version: "encryptionkey@0"
I20260927 04:07:06.985636 6550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260927 04:07:06.986495 6565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:06.986696 6550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:07:06.986804 6550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "1fe2444dc8af45129a6bbaa55fa42051"
format_stamp: "Formatted at 2026-09-27 04:07:06 on dist-test-slave-zqrn"
server_key: "d34bdd7eead8450c32f921eb1d6f865f"
server_key_iv: "687e6f394b949121dcf979ef9f107c37"
server_key_version: "encryptionkey@0"
I20260927 04:07:06.986896 6550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:07.003377 6550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:07.005275 6550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:07.005445 6550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:07.009508 6550 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38457
I20260927 04:07:07.009549 6617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38457 every 8 connection(s)
I20260927 04:07:07.009919 6550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:07.010764 6618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:07.013037 6618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051: Bootstrap starting.
I20260927 04:07:07.013616 6618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:07.013911 6618 log.cc:824] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:07.014613 6618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051: No bootstrap required, opened a new log
I20260927 04:07:07.016206 6618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2444dc8af45129a6bbaa55fa42051" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38457 } }
I20260927 04:07:07.016340 6618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:07.016362 6618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fe2444dc8af45129a6bbaa55fa42051, State: Initialized, Role: FOLLOWER
I20260927 04:07:07.016517 6618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2444dc8af45129a6bbaa55fa42051" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38457 } }
I20260927 04:07:07.016579 6618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:07.016600 6618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:07.016646 6618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:07.017199 6618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2444dc8af45129a6bbaa55fa42051" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38457 } }
I20260927 04:07:07.017303 6618 leader_election.cc:302] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fe2444dc8af45129a6bbaa55fa42051; no voters:
I20260927 04:07:07.017452 6618 leader_election.cc:288] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:07.017500 6621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:07.017736 6621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [term 1 LEADER]: Becoming Leader. State: Replica: 1fe2444dc8af45129a6bbaa55fa42051, State: Running, Role: LEADER
I20260927 04:07:07.017759 6618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:07.017880 6621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2444dc8af45129a6bbaa55fa42051" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38457 } }
I20260927 04:07:07.018589 6622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fe2444dc8af45129a6bbaa55fa42051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2444dc8af45129a6bbaa55fa42051" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38457 } } }
I20260927 04:07:07.018749 6622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:07.018919 6631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fe2444dc8af45129a6bbaa55fa42051. Latest consensus state: current_term: 1 leader_uuid: "1fe2444dc8af45129a6bbaa55fa42051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fe2444dc8af45129a6bbaa55fa42051" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38457 } } }
I20260927 04:07:07.019086 6631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:07.019153 6632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:07.019608 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6550
I20260927 04:07:07.019733 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:07.019925 18147 external_mini_cluster.cc:1428] Setting key f961f754c0f26f2618d30bc13745ac75
I20260927 04:07:07.019928 6632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:07.021705 6632 catalog_manager.cc:1347] Generated new cluster ID: b2b07e2470d24c09afc9ac82e851be8d
I20260927 04:07:07.021801 6632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:07:07 dist-test-slave-zqrn krb5kdc[6537](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 04:07:07.034307 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:07.020365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:39579 (local address 127.0.0.1:38457)
0927 04:07:07.023011 (+ 2646us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:07.023020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:07.023235 (+ 215us) server_negotiation.cc:407] Connection header received
0927 04:07:07.023372 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:07.023380 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:07.023448 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:07.023609 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:07.024316 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:07.025774 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:07.026788 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:07.027470 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:07.027591 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:07.030390 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:07.030394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:07.030394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:07.030408 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:07.032466 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:07.032951 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:07.032954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:07.032954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:07.033058 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:07.033286 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:07.033286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:07.033287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:07.033653 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:07.033797 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:07.033962 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 04:07:07.034055 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2595,"thread_start_us":65,"threads_started":1}
I20260927 04:07:07.035626 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:07.044549 6641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:07.037079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:46613 (local address 127.0.0.1:38457)
0927 04:07:07.037129 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:07.037131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:07.037201 (+ 70us) server_negotiation.cc:407] Connection header received
0927 04:07:07.037277 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:07.037278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:07.037288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:07.037333 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:07.038003 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:07.038724 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:07.039837 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:07.040155 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:07.040209 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:07.041686 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:07.041687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:07.041688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:07.041694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:07.043301 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:07.043724 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:07.043725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:07.043725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:07.043780 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:07.044005 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:07.044006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:07.044006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:07.044132 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:07.044243 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:07.044390 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 04:07:07.044454 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:07:07.057832 6632 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:07.058354 6632 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:07.067487 6632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fe2444dc8af45129a6bbaa55fa42051: Generated new TSK 0
I20260927 04:07:07.067698 6632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:07.074182 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6550
W20260927 04:07:07.078572 6535 connection.cc:535] client connection to 127.0.0.1:38457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38457): Connection reset by peer (error 104)
2026-09-27T04:07:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:07 dist-test-slave-zqrn krb5kdc[6650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:07 dist-test-slave-zqrn krb5kdc[6650](info): set up 2 sockets
Sep 27 04:07:07 dist-test-slave-zqrn krb5kdc[6650](info): commencing operation
krb5kdc: starting...
W20260927 04:07:08.119149 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:08 dist-test-slave-zqrn krb5kdc[6650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:08.247146 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:08.331275 6663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:08.331574 6663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:08.331621 6663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:08.332863 6663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:08.332957 6663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:08.332983 6663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:08.333004 6663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:08.333030 6663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:08.334519 6663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:08.334807 6663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:08.335106 6663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:08.340344 6668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:08.340350 6669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:08.340380 6663 server_base.cc:1034] running on GCE node
W20260927 04:07:08.340368 6671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:08.340936 6663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:08.341200 6663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:08.342355 6663 hybrid_clock.cc:648] HybridClock initialized: now 1790482028342314 us; error 54 us; skew 500 ppm
Sep 27 04:07:08 dist-test-slave-zqrn krb5kdc[6650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:08.345094 6663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:08.345571 6663 webserver.cc:460] Webserver started at http://127.0.0.1:39827/ using document root <none> and password file <none>
I20260927 04:07:08.345805 6663 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:08.345852 6663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:08.346006 6663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:08.347182 6663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "ea759bdd7659459d9fc942cd1c81ee4d"
format_stamp: "Formatted at 2026-09-27 04:07:08 on dist-test-slave-zqrn"
server_key: "2d932559cb67bb0d68bb132a49475141"
server_key_iv: "20f6567329eb7b1d34f727bc2bfcddf0"
server_key_version: "encryptionkey@0"
I20260927 04:07:08.347546 6663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "ea759bdd7659459d9fc942cd1c81ee4d"
format_stamp: "Formatted at 2026-09-27 04:07:08 on dist-test-slave-zqrn"
server_key: "2d932559cb67bb0d68bb132a49475141"
server_key_iv: "20f6567329eb7b1d34f727bc2bfcddf0"
server_key_version: "encryptionkey@0"
I20260927 04:07:08.348886 6663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:08.349648 6678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:08.349865 6663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:08.349982 6663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "ea759bdd7659459d9fc942cd1c81ee4d"
format_stamp: "Formatted at 2026-09-27 04:07:08 on dist-test-slave-zqrn"
server_key: "2d932559cb67bb0d68bb132a49475141"
server_key_iv: "20f6567329eb7b1d34f727bc2bfcddf0"
server_key_version: "encryptionkey@0"
I20260927 04:07:08.350061 6663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:08.399482 6663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:08.401937 6663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:08.402117 6663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:08.406136 6663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35277
I20260927 04:07:08.406176 6730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35277 every 8 connection(s)
I20260927 04:07:08.406569 6663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:08.407619 6731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:08.409858 6731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d: Bootstrap starting.
I20260927 04:07:08.410384 6731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:08.410625 6731 log.cc:824] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:08.411334 6731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d: No bootstrap required, opened a new log
I20260927 04:07:08.412873 6731 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea759bdd7659459d9fc942cd1c81ee4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } }
I20260927 04:07:08.413105 6731 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:08.413123 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6663
I20260927 04:07:08.413146 6731 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea759bdd7659459d9fc942cd1c81ee4d, State: Initialized, Role: FOLLOWER
I20260927 04:07:08.413228 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:08.413298 6731 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea759bdd7659459d9fc942cd1c81ee4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } }
I20260927 04:07:08.413386 6731 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:08.413414 18147 external_mini_cluster.cc:1428] Setting key 07b90f73e14d912742913900636d7b6b
I20260927 04:07:08.413415 6731 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:08.413463 6731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:08.414572 6731 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea759bdd7659459d9fc942cd1c81ee4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } }
I20260927 04:07:08.414696 6731 leader_election.cc:302] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea759bdd7659459d9fc942cd1c81ee4d; no voters:
I20260927 04:07:08.414865 6731 leader_election.cc:288] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:08.414912 6736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:08.415079 6736 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [term 1 LEADER]: Becoming Leader. State: Replica: ea759bdd7659459d9fc942cd1c81ee4d, State: Running, Role: LEADER
I20260927 04:07:08.415138 6731 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:08.415218 6736 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea759bdd7659459d9fc942cd1c81ee4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } }
I20260927 04:07:08.415714 6737 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea759bdd7659459d9fc942cd1c81ee4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea759bdd7659459d9fc942cd1c81ee4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } } }
I20260927 04:07:08.415802 6737 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:08.415699 6738 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea759bdd7659459d9fc942cd1c81ee4d. Latest consensus state: current_term: 1 leader_uuid: "ea759bdd7659459d9fc942cd1c81ee4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea759bdd7659459d9fc942cd1c81ee4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35277 } } }
I20260927 04:07:08.416039 6738 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:08.416400 6745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:08.416870 6745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:08.418411 6745 catalog_manager.cc:1347] Generated new cluster ID: 2d952828de1d402b88a64cded8f6d42a
I20260927 04:07:08.418497 6745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:08.419430 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:08.413931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:43263 (local address 127.0.0.1:35277)
0927 04:07:08.414196 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:08.414203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:08.414228 (+ 25us) server_negotiation.cc:407] Connection header received
0927 04:07:08.414321 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:08.414326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:08.414369 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:08.414496 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:08.414750 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:08.416186 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:08.417293 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:08.417868 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:08.417977 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:08.418728 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:08.418732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:08.418732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:08.418965 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:08.419003 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:08.419017 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:08.419119 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:08.419199 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 04:07:08.419228 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":158,"threads_started":1}
I20260927 04:07:08.420424 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:08.425758 6735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:08.421883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:50887 (local address 127.0.0.1:35277)
0927 04:07:08.421918 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:08.421920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:08.422059 (+ 139us) server_negotiation.cc:407] Connection header received
0927 04:07:08.422164 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:08.422165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:08.422175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:08.422204 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:08.422396 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:08.423183 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:08.424301 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:08.424638 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:08.424682 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:08.425484 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:08.425485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:08.425486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:08.425490 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:08.425503 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:08.425507 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:08.425539 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:08.425670 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 04:07:08.425683 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260927 04:07:08.427649 6745 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:08.428145 6745 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:08.440917 6745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea759bdd7659459d9fc942cd1c81ee4d: Generated new TSK 0
I20260927 04:07:08.441149 6745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:08.449828 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6663
W20260927 04:07:08.454172 6648 connection.cc:535] client connection to 127.0.0.1:35277 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35277): Connection reset by peer (error 104)
2026-09-27T04:07:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:08 dist-test-slave-zqrn krb5kdc[6762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:08 dist-test-slave-zqrn krb5kdc[6762](info): set up 2 sockets
Sep 27 04:07:08 dist-test-slave-zqrn krb5kdc[6762](info): commencing operation
krb5kdc: starting...
W20260927 04:07:09.523553 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:09 dist-test-slave-zqrn krb5kdc[6762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:09.645730 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:09.732051 6775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:09.732293 6775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:09.732319 6775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:09.733536 6775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:09.733585 6775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:09.733603 6775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:09.733619 6775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:09.733633 6775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:09.735177 6775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:09.735467 6775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:09.735718 6775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:09.740841 6781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:09.740855 6783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:09.740856 6780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:09.741174 6775 server_base.cc:1034] running on GCE node
I20260927 04:07:09.741408 6775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:09.741704 6775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:09.742885 6775 hybrid_clock.cc:648] HybridClock initialized: now 1790482029742872 us; error 45 us; skew 500 ppm
Sep 27 04:07:09 dist-test-slave-zqrn krb5kdc[6762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:09.751489 6775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:09.752009 6775 webserver.cc:460] Webserver started at http://127.0.0.1:43257/ using document root <none> and password file <none>
I20260927 04:07:09.752285 6775 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:09.752362 6775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:09.752514 6775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:09.753602 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "387f326e244446d586ec76f2bed263af"
format_stamp: "Formatted at 2026-09-27 04:07:09 on dist-test-slave-zqrn"
server_key: "e2a3158a841c71ae5becda0ad1cc5a83"
server_key_iv: "e8dee348b8ec91c9905297ff3ec0bda1"
server_key_version: "encryptionkey@0"
I20260927 04:07:09.753978 6775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "387f326e244446d586ec76f2bed263af"
format_stamp: "Formatted at 2026-09-27 04:07:09 on dist-test-slave-zqrn"
server_key: "e2a3158a841c71ae5becda0ad1cc5a83"
server_key_iv: "e8dee348b8ec91c9905297ff3ec0bda1"
server_key_version: "encryptionkey@0"
I20260927 04:07:09.755421 6775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:07:09.756196 6790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:09.756403 6775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:07:09.756510 6775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "387f326e244446d586ec76f2bed263af"
format_stamp: "Formatted at 2026-09-27 04:07:09 on dist-test-slave-zqrn"
server_key: "e2a3158a841c71ae5becda0ad1cc5a83"
server_key_iv: "e8dee348b8ec91c9905297ff3ec0bda1"
server_key_version: "encryptionkey@0"
I20260927 04:07:09.756585 6775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:09.760054 6775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:09.762315 6775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:09.762475 6775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:09.766695 6775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45733
I20260927 04:07:09.766738 6842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45733 every 8 connection(s)
I20260927 04:07:09.767132 6775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:09.768029 6843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:09.770144 6843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af: Bootstrap starting.
I20260927 04:07:09.770644 6843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:09.770936 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6775
I20260927 04:07:09.770884 6843 log.cc:824] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:09.771064 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:09.771271 18147 external_mini_cluster.cc:1428] Setting key c8893fa0ae365b8471c6f020fbe670a9
I20260927 04:07:09.771631 6843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af: No bootstrap required, opened a new log
I20260927 04:07:09.773964 6843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f326e244446d586ec76f2bed263af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45733 } }
I20260927 04:07:09.774143 6843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:09.774191 6843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387f326e244446d586ec76f2bed263af, State: Initialized, Role: FOLLOWER
I20260927 04:07:09.774319 6843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f326e244446d586ec76f2bed263af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45733 } }
I20260927 04:07:09.774370 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:09.771779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:43287 (local address 127.0.0.1:45733)
0927 04:07:09.771950 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:09.771963 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:09.771987 (+ 24us) server_negotiation.cc:407] Connection header received
0927 04:07:09.772923 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:09.772931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:09.772957 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:09.773072 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:09.773705 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:09.773708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:09.773709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:09.773712 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:09.773747 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:09.773757 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:09.773812 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:09.773897 (+ 85us) server_negotiation.cc:299] Negotiation successful
0927 04:07:09.773956 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260927 04:07:09.774422 6843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:09.774464 6843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:09.774514 6843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:09.775146 6843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f326e244446d586ec76f2bed263af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45733 } }
I20260927 04:07:09.775261 6843 leader_election.cc:302] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 387f326e244446d586ec76f2bed263af; no voters:
I20260927 04:07:09.775290 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:09.775413 6843 leader_election.cc:288] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:09.775465 6848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:09.775631 6843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:09.775642 6848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [term 1 LEADER]: Becoming Leader. State: Replica: 387f326e244446d586ec76f2bed263af, State: Running, Role: LEADER
I20260927 04:07:09.775750 6848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f326e244446d586ec76f2bed263af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45733 } }
I20260927 04:07:09.776018 6848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 387f326e244446d586ec76f2bed263af. Latest consensus state: current_term: 1 leader_uuid: "387f326e244446d586ec76f2bed263af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f326e244446d586ec76f2bed263af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45733 } } }
I20260927 04:07:09.776077 6848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:09.776320 6849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "387f326e244446d586ec76f2bed263af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "387f326e244446d586ec76f2bed263af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45733 } } }
I20260927 04:07:09.776434 6849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:09.776989 6852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:09.777468 6852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:09.778970 6852 catalog_manager.cc:1347] Generated new cluster ID: 4905690ecd5f4b47b9daffaf6855aa4f
I20260927 04:07:09.779031 6852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:09.779428 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:09.778174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:33993 (local address 127.0.0.1:45733)
0927 04:07:09.778205 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:09.778207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:09.778329 (+ 122us) server_negotiation.cc:407] Connection header received
0927 04:07:09.778379 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:09.778379 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:09.778383 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:09.778444 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:09.779213 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:09.779214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:09.779214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:09.779218 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:09.779232 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:09.779235 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:09.779285 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:09.779351 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 04:07:09.779366 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260927 04:07:09.794813 6852 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:09.795001 6852 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:09.806172 6852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 387f326e244446d586ec76f2bed263af: Generated new TSK 0
I20260927 04:07:09.806421 6852 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:09.812700 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6775
2026-09-27T04:07:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:09 dist-test-slave-zqrn krb5kdc[6874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:09 dist-test-slave-zqrn krb5kdc[6874](info): set up 2 sockets
Sep 27 04:07:09 dist-test-slave-zqrn krb5kdc[6874](info): commencing operation
krb5kdc: starting...
W20260927 04:07:10.867177 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:10 dist-test-slave-zqrn krb5kdc[6874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:10.993633 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:11.076992 6887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:11.077211 6887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:11.077235 6887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:11.078507 6887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:11.078552 6887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:11.078565 6887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:11.078578 6887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:11.078590 6887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:11.080132 6887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37073
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:11.080396 6887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:11.080636 6887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:11.085667 6893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:11.085791 6895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:11.086019 6887 server_base.cc:1034] running on GCE node
W20260927 04:07:11.085722 6892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:11.086346 6887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:11.086655 6887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:11.087800 6887 hybrid_clock.cc:648] HybridClock initialized: now 1790482031087761 us; error 51 us; skew 500 ppm
Sep 27 04:07:11 dist-test-slave-zqrn krb5kdc[6874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:11.090337 6887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:11.090811 6887 webserver.cc:460] Webserver started at http://127.0.0.1:33021/ using document root <none> and password file <none>
I20260927 04:07:11.091074 6887 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:11.091127 6887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:11.091225 6887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:11.092247 6887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "801714cc9f024c78897d091b8e0a70e1"
format_stamp: "Formatted at 2026-09-27 04:07:11 on dist-test-slave-zqrn"
server_key: "d827262175b928e5b5eab29aa288f7d0"
server_key_iv: "e2c4611b2ed66b6c5cc3b994679d605e"
server_key_version: "encryptionkey@0"
I20260927 04:07:11.092537 6887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "801714cc9f024c78897d091b8e0a70e1"
format_stamp: "Formatted at 2026-09-27 04:07:11 on dist-test-slave-zqrn"
server_key: "d827262175b928e5b5eab29aa288f7d0"
server_key_iv: "e2c4611b2ed66b6c5cc3b994679d605e"
server_key_version: "encryptionkey@0"
I20260927 04:07:11.093825 6887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:07:11.094537 6902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:11.094717 6887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:07:11.094816 6887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "801714cc9f024c78897d091b8e0a70e1"
format_stamp: "Formatted at 2026-09-27 04:07:11 on dist-test-slave-zqrn"
server_key: "d827262175b928e5b5eab29aa288f7d0"
server_key_iv: "e2c4611b2ed66b6c5cc3b994679d605e"
server_key_version: "encryptionkey@0"
I20260927 04:07:11.094897 6887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:11.105458 6887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:11.107508 6887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:11.107676 6887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:11.111666 6887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37073
I20260927 04:07:11.111709 6954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37073 every 8 connection(s)
I20260927 04:07:11.112080 6887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:11.113046 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:11.115677 6955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1: Bootstrap starting.
I20260927 04:07:11.116274 6955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:11.116555 6955 log.cc:824] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:11.117270 6955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1: No bootstrap required, opened a new log
I20260927 04:07:11.118517 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6887
I20260927 04:07:11.118654 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:11.118883 18147 external_mini_cluster.cc:1428] Setting key f20d0c0b5f9302cf9fc098b088a2ddfa
I20260927 04:07:11.119813 6955 raft_consensus.cc:348] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801714cc9f024c78897d091b8e0a70e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260927 04:07:11.120004 6955 raft_consensus.cc:374] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:11.120049 6955 raft_consensus.cc:729] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801714cc9f024c78897d091b8e0a70e1, State: Initialized, Role: FOLLOWER
I20260927 04:07:11.120265 6955 consensus_queue.cc:260] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801714cc9f024c78897d091b8e0a70e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260927 04:07:11.120376 6955 raft_consensus.cc:388] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:11.120420 6955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:11.120497 6955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:11.121702 6955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801714cc9f024c78897d091b8e0a70e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260927 04:07:11.121802 6955 leader_election.cc:302] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 801714cc9f024c78897d091b8e0a70e1; no voters:
I20260927 04:07:11.122040 6955 leader_election.cc:288] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:11.122087 6960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:11.122277 6960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [term 1 LEADER]: Becoming Leader. State: Replica: 801714cc9f024c78897d091b8e0a70e1, State: Running, Role: LEADER
I20260927 04:07:11.122411 6960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801714cc9f024c78897d091b8e0a70e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } }
I20260927 04:07:11.122418 6955 sys_catalog.cc:564] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:11.123374 6962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 801714cc9f024c78897d091b8e0a70e1. Latest consensus state: current_term: 1 leader_uuid: "801714cc9f024c78897d091b8e0a70e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801714cc9f024c78897d091b8e0a70e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } } }
I20260927 04:07:11.123489 6962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:11.124157 6961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "801714cc9f024c78897d091b8e0a70e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "801714cc9f024c78897d091b8e0a70e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37073 } } }
I20260927 04:07:11.124357 6961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1 [sys.catalog]: This master's current role is: LEADER
W20260927 04:07:11.124483 6976 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:11.124549 6976 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:11.124641 6970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:11.124806 6970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:11.126159 6970 catalog_manager.cc:1347] Generated new cluster ID: 0473fc92193442dfa7bb211fb6e3a49c
I20260927 04:07:11.126206 6970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:07:11 dist-test-slave-zqrn krb5kdc[6874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 04:07:11.131675 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:11.119481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:50123 (local address 127.0.0.1:37073)
0927 04:07:11.119694 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:11.119701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:11.119730 (+ 29us) server_negotiation.cc:407] Connection header received
0927 04:07:11.119826 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:11.119832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:11.119876 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:11.119985 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:11.120992 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:11.122003 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:11.123250 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:11.123918 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:11.124034 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:11.127628 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:11.127632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:11.127632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:11.127657 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:11.129619 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:11.130129 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:11.130132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:11.130132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:11.130249 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:11.130551 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:11.130551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:11.130552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:11.130835 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:11.131021 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:11.131190 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 04:07:11.131283 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260927 04:07:11.132772 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:11.142042 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:11.134336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:37127 (local address 127.0.0.1:37073)
0927 04:07:11.134376 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:11.134378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:11.134406 (+ 28us) server_negotiation.cc:407] Connection header received
0927 04:07:11.134429 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:11.134429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:11.134454 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:11.134515 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:11.135133 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:11.135928 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:11.136932 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:11.137265 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:11.137356 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:11.138710 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:11.138712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:11.138712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:11.138720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:11.140613 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:11.141035 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:11.141036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:11.141036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:11.141106 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:11.141347 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:11.141348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:11.141348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:11.141505 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:11.141642 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:11.141743 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 04:07:11.141815 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:07:11.149228 6970 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:11.149750 6970 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:11.154737 6970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 801714cc9f024c78897d091b8e0a70e1: Generated new TSK 0
I20260927 04:07:11.154922 6970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:11.157665 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6887
W20260927 04:07:11.162228 6872 connection.cc:535] client connection to 127.0.0.1:37073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37073): Connection reset by peer (error 104)
2026-09-27T04:07:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:11 dist-test-slave-zqrn krb5kdc[6986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:11 dist-test-slave-zqrn krb5kdc[6986](info): set up 2 sockets
Sep 27 04:07:11 dist-test-slave-zqrn krb5kdc[6986](info): commencing operation
krb5kdc: starting...
W20260927 04:07:12.227197 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:12 dist-test-slave-zqrn krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:12.349607 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43185
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:12.432183 6999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:12.432432 6999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:12.432471 6999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:12.433641 6999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:12.433709 6999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:12.433735 6999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:12.433763 6999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:12.433781 6999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:12.435286 6999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43185
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:12.435554 6999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:12.435770 6999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:12.440733 7004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:12.440724 7005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:12.440742 6999 server_base.cc:1034] running on GCE node
W20260927 04:07:12.440724 7007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:12.441190 6999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:12.441431 6999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:12.442576 6999 hybrid_clock.cc:648] HybridClock initialized: now 1790482032442563 us; error 30 us; skew 500 ppm
Sep 27 04:07:12 dist-test-slave-zqrn krb5kdc[6986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:12.445429 6999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:12.445861 6999 webserver.cc:460] Webserver started at http://127.0.0.1:36131/ using document root <none> and password file <none>
I20260927 04:07:12.446096 6999 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:12.446148 6999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:12.446314 6999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:12.447484 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "a7210ca7a955456495fbd52f896f0eb0"
format_stamp: "Formatted at 2026-09-27 04:07:12 on dist-test-slave-zqrn"
server_key: "e9c7443b1ae18648bcdcfc8349bfa5c4"
server_key_iv: "0f631524fb06c60a70c880e9e8f7d340"
server_key_version: "encryptionkey@0"
I20260927 04:07:12.447847 6999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "a7210ca7a955456495fbd52f896f0eb0"
format_stamp: "Formatted at 2026-09-27 04:07:12 on dist-test-slave-zqrn"
server_key: "e9c7443b1ae18648bcdcfc8349bfa5c4"
server_key_iv: "0f631524fb06c60a70c880e9e8f7d340"
server_key_version: "encryptionkey@0"
I20260927 04:07:12.449329 6999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:12.450091 7014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:12.450281 6999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:12.450372 6999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "a7210ca7a955456495fbd52f896f0eb0"
format_stamp: "Formatted at 2026-09-27 04:07:12 on dist-test-slave-zqrn"
server_key: "e9c7443b1ae18648bcdcfc8349bfa5c4"
server_key_iv: "0f631524fb06c60a70c880e9e8f7d340"
server_key_version: "encryptionkey@0"
I20260927 04:07:12.450455 6999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:12.474457 6999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:12.476969 6999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:12.477090 6999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:12.481148 6999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43185
I20260927 04:07:12.481186 7066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43185 every 8 connection(s)
I20260927 04:07:12.481576 6999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:12.482611 7067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:12.484874 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 6999
I20260927 04:07:12.485014 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:12.485175 7067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0: Bootstrap starting.
I20260927 04:07:12.485216 18147 external_mini_cluster.cc:1428] Setting key c3ed6e1130cbac6296f6d6a963958fee
I20260927 04:07:12.485975 7067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:12.486297 7067 log.cc:824] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:12.487133 7067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0: No bootstrap required, opened a new log
I20260927 04:07:12.489053 7067 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7210ca7a955456495fbd52f896f0eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43185 } }
I20260927 04:07:12.489244 7067 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:12.489274 7067 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7210ca7a955456495fbd52f896f0eb0, State: Initialized, Role: FOLLOWER
I20260927 04:07:12.489432 7067 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7210ca7a955456495fbd52f896f0eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43185 } }
I20260927 04:07:12.489500 7067 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:12.489524 7067 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:12.489598 7067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:12.490402 7067 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7210ca7a955456495fbd52f896f0eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43185 } }
I20260927 04:07:12.490520 7067 leader_election.cc:302] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7210ca7a955456495fbd52f896f0eb0; no voters:
I20260927 04:07:12.490913 7067 leader_election.cc:288] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:12.490940 7072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:12.491117 7072 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [term 1 LEADER]: Becoming Leader. State: Replica: a7210ca7a955456495fbd52f896f0eb0, State: Running, Role: LEADER
I20260927 04:07:12.491210 7072 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7210ca7a955456495fbd52f896f0eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43185 } }
I20260927 04:07:12.491215 7067 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:12.491288 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:12.485838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:51437 (local address 127.0.0.1:43185)
0927 04:07:12.486025 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:12.486034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:12.486069 (+ 35us) server_negotiation.cc:407] Connection header received
0927 04:07:12.486622 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:12.486629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:12.486685 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:12.486852 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:12.487106 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:12.488057 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:12.488878 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:12.489473 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:12.489611 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:12.490372 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:12.490376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:12.490376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:12.490613 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:12.490653 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:12.490684 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:12.490809 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:12.491026 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 04:07:12.491058 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260927 04:07:12.492453 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:12.492458 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7210ca7a955456495fbd52f896f0eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7210ca7a955456495fbd52f896f0eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43185 } } }
I20260927 04:07:12.492540 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:12.492753 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7210ca7a955456495fbd52f896f0eb0. Latest consensus state: current_term: 1 leader_uuid: "a7210ca7a955456495fbd52f896f0eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7210ca7a955456495fbd52f896f0eb0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43185 } } }
I20260927 04:07:12.492851 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0 [sys.catalog]: This master's current role is: LEADER
W20260927 04:07:12.493188 7086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:12.493238 7086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:12.493299 7087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:12.493501 7087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:12.495254 7087 catalog_manager.cc:1347] Generated new cluster ID: 057bb1a4959c4f0680b7d26c337c43d9
I20260927 04:07:12.495309 7087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:12.498380 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:12.494653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:41685 (local address 127.0.0.1:43185)
0927 04:07:12.494694 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:12.494697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:12.494921 (+ 224us) server_negotiation.cc:407] Connection header received
0927 04:07:12.495004 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:12.495004 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:12.495015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:12.495079 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:12.495248 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:12.495970 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:12.496993 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:12.497326 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:12.497401 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:12.498120 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:12.498121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:12.498122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:12.498126 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:12.498142 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:12.498146 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:12.498212 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:12.498293 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 04:07:12.498308 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260927 04:07:12.517848 7087 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:12.518426 7087 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:12.528760 7087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7210ca7a955456495fbd52f896f0eb0: Generated new TSK 0
I20260927 04:07:12.528975 7087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:12.538508 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 6999
W20260927 04:07:12.543110 6984 connection.cc:535] client connection to 127.0.0.1:43185 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43185): Connection reset by peer (error 104)
2026-09-27T04:07:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:12 dist-test-slave-zqrn krb5kdc[7098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:12 dist-test-slave-zqrn krb5kdc[7098](info): set up 2 sockets
Sep 27 04:07:12 dist-test-slave-zqrn krb5kdc[7098](info): commencing operation
krb5kdc: starting...
W20260927 04:07:13.635143 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:13 dist-test-slave-zqrn krb5kdc[7098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 04:07:13.766782 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:13.852376 7111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:13.852658 7111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:13.852697 7111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:13.853873 7111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:13.853963 7111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:13.853995 7111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:13.854029 7111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:13.854053 7111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:13.855571 7111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:13.855867 7111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:13.856120 7111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:13.861517 7116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:13.861529 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:13.861537 7119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:13.861905 7111 server_base.cc:1034] running on GCE node
I20260927 04:07:13.862145 7111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:13.862401 7111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:13.863539 7111 hybrid_clock.cc:648] HybridClock initialized: now 1790482033863526 us; error 30 us; skew 500 ppm
Sep 27 04:07:13 dist-test-slave-zqrn krb5kdc[7098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:13.866206 7111 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 04:07:13.866660 7111 webserver.cc:460] Webserver started at http://127.0.0.1:32837/ using document root <none> and password file <none>
I20260927 04:07:13.866932 7111 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:13.867009 7111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:13.867167 7111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:13.868283 7111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "24b07fd57e2b41efb6a1817e91cf840d"
format_stamp: "Formatted at 2026-09-27 04:07:13 on dist-test-slave-zqrn"
server_key: "7b8db53381b262ef458ef7f22cd307dc"
server_key_iv: "2b4994f3543b7842a55a0c84bf26ec3e"
server_key_version: "encryptionkey@0"
I20260927 04:07:13.868659 7111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "24b07fd57e2b41efb6a1817e91cf840d"
format_stamp: "Formatted at 2026-09-27 04:07:13 on dist-test-slave-zqrn"
server_key: "7b8db53381b262ef458ef7f22cd307dc"
server_key_iv: "2b4994f3543b7842a55a0c84bf26ec3e"
server_key_version: "encryptionkey@0"
I20260927 04:07:13.870023 7111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:07:13.870862 7126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:13.871111 7111 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:13.871214 7111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "24b07fd57e2b41efb6a1817e91cf840d"
format_stamp: "Formatted at 2026-09-27 04:07:13 on dist-test-slave-zqrn"
server_key: "7b8db53381b262ef458ef7f22cd307dc"
server_key_iv: "2b4994f3543b7842a55a0c84bf26ec3e"
server_key_version: "encryptionkey@0"
I20260927 04:07:13.871296 7111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:13.874708 7111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:13.876878 7111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:13.877039 7111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:13.881731 7111 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39983
I20260927 04:07:13.881760 7178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39983 every 8 connection(s)
I20260927 04:07:13.882122 7111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:13.883131 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:13.885602 7179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d: Bootstrap starting.
I20260927 04:07:13.886253 7179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:13.886562 7179 log.cc:824] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:13.887427 7179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d: No bootstrap required, opened a new log
I20260927 04:07:13.889199 7179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b07fd57e2b41efb6a1817e91cf840d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39983 } }
I20260927 04:07:13.889348 7179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:13.889374 7179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b07fd57e2b41efb6a1817e91cf840d, State: Initialized, Role: FOLLOWER
I20260927 04:07:13.889550 7179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b07fd57e2b41efb6a1817e91cf840d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39983 } }
I20260927 04:07:13.889628 7179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:13.889653 7179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:13.889726 7179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:13.890332 7179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b07fd57e2b41efb6a1817e91cf840d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39983 } }
I20260927 04:07:13.890468 7179 leader_election.cc:302] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24b07fd57e2b41efb6a1817e91cf840d; no voters:
I20260927 04:07:13.890676 7179 leader_election.cc:288] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:13.890743 7182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:13.890988 7179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:13.891000 7182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [term 1 LEADER]: Becoming Leader. State: Replica: 24b07fd57e2b41efb6a1817e91cf840d, State: Running, Role: LEADER
I20260927 04:07:13.891218 7182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b07fd57e2b41efb6a1817e91cf840d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39983 } }
I20260927 04:07:13.891638 7184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24b07fd57e2b41efb6a1817e91cf840d. Latest consensus state: current_term: 1 leader_uuid: "24b07fd57e2b41efb6a1817e91cf840d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b07fd57e2b41efb6a1817e91cf840d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39983 } } }
I20260927 04:07:13.891736 7184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:13.891770 7183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24b07fd57e2b41efb6a1817e91cf840d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24b07fd57e2b41efb6a1817e91cf840d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39983 } } }
I20260927 04:07:13.891831 7183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:13.891976 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7111
I20260927 04:07:13.892102 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:13.892158 7191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:13.892383 18147 external_mini_cluster.cc:1428] Setting key 51a79f19ab9848c56fa4ddd806f92df6
I20260927 04:07:13.892763 7191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:13.894699 7191 catalog_manager.cc:1347] Generated new cluster ID: 8a4010f72c16496fa7787193c2a05e83
I20260927 04:07:13.894758 7191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:13.899896 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:13.893407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:59863 (local address 127.0.0.1:39983)
0927 04:07:13.893603 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:13.893612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:13.893649 (+ 37us) server_negotiation.cc:407] Connection header received
0927 04:07:13.893773 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:13.893781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:13.893812 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:13.893968 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:13.899214 (+ 5246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:13.899219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:13.899219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:13.899226 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:13.899280 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:13.899297 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:13.899390 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:13.899477 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 04:07:13.899527 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":114,"threads_started":1}
I20260927 04:07:13.900936 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:13.910883 7191 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:13.911101 7191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:13.912055 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:13.902546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.209:60915 (local address 127.0.0.1:39983)
0927 04:07:13.902584 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:13.902586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:13.910852 (+ 8266us) server_negotiation.cc:407] Connection header received
0927 04:07:13.910959 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:13.910960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:13.910966 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:13.911043 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:13.911807 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:13.911809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:13.911809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:13.911815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:13.911833 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:13.911839 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:13.911908 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:13.911969 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 04:07:13.911985 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:07:13.925427 7191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24b07fd57e2b41efb6a1817e91cf840d: Generated new TSK 0
I20260927 04:07:13.925648 7191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:13.928558 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7111
2026-09-27T04:07:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:13 dist-test-slave-zqrn krb5kdc[7210](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:13 dist-test-slave-zqrn krb5kdc[7210](info): set up 2 sockets
Sep 27 04:07:13 dist-test-slave-zqrn krb5kdc[7210](info): commencing operation
krb5kdc: starting...
W20260927 04:07:14.975206 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:15 dist-test-slave-zqrn krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:07:15.102818 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:40635
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:15.183367 7223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:15.183642 7223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:15.183683 7223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:15.184834 7223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:15.184913 7223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:15.184937 7223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:15.184959 7223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:15.184981 7223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:15.186440 7223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:40635
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:15.186722 7223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:15.186967 7223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:15.192018 7231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:15.191983 7229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:15.192004 7228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:15.192612 7223 server_base.cc:1034] running on GCE node
I20260927 04:07:15.192840 7223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:15.193112 7223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:15.194269 7223 hybrid_clock.cc:648] HybridClock initialized: now 1790482035194251 us; error 42 us; skew 500 ppm
Sep 27 04:07:15 dist-test-slave-zqrn krb5kdc[7210](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482035, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:15.201944 7223 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:07:15.202493 7223 webserver.cc:460] Webserver started at http://127.17.184.254:43455/ using document root <none> and password file <none>
I20260927 04:07:15.202811 7223 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:15.202893 7223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:15.203079 7223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:15.204265 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "6bfedf1c8a2747a3b8fd21758c541896"
format_stamp: "Formatted at 2026-09-27 04:07:15 on dist-test-slave-zqrn"
server_key: "5639f7351fc5bea9c70811493423aeb5"
server_key_iv: "7715b7c7581a32a081ef1c562f5d5a75"
server_key_version: "encryptionkey@0"
I20260927 04:07:15.204674 7223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "6bfedf1c8a2747a3b8fd21758c541896"
format_stamp: "Formatted at 2026-09-27 04:07:15 on dist-test-slave-zqrn"
server_key: "5639f7351fc5bea9c70811493423aeb5"
server_key_iv: "7715b7c7581a32a081ef1c562f5d5a75"
server_key_version: "encryptionkey@0"
I20260927 04:07:15.206205 7223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:07:15.207175 7238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:15.207430 7223 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:15.207559 7223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "6bfedf1c8a2747a3b8fd21758c541896"
format_stamp: "Formatted at 2026-09-27 04:07:15 on dist-test-slave-zqrn"
server_key: "5639f7351fc5bea9c70811493423aeb5"
server_key_iv: "7715b7c7581a32a081ef1c562f5d5a75"
server_key_version: "encryptionkey@0"
I20260927 04:07:15.207659 7223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:15.228957 7223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:15.231161 7223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:15.231274 7223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:15.235183 7223 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:40635
I20260927 04:07:15.235216 7290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:40635 every 8 connection(s)
I20260927 04:07:15.235584 7223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:15.236505 7291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:15.238243 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7223
I20260927 04:07:15.238390 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:15.238579 18147 external_mini_cluster.cc:1428] Setting key 7c13dd1f35ef9483ed223b631e09849f
I20260927 04:07:15.238976 7291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896: Bootstrap starting.
I20260927 04:07:15.239640 7291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:15.239939 7291 log.cc:824] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:15.240684 7291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896: No bootstrap required, opened a new log
I20260927 04:07:15.242435 7291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfedf1c8a2747a3b8fd21758c541896" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40635 } }
I20260927 04:07:15.242609 7291 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:15.242647 7291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bfedf1c8a2747a3b8fd21758c541896, State: Initialized, Role: FOLLOWER
I20260927 04:07:15.242799 7291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfedf1c8a2747a3b8fd21758c541896" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40635 } }
I20260927 04:07:15.242883 7291 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:15.242947 7291 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:15.242996 7291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:15.243619 7291 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfedf1c8a2747a3b8fd21758c541896" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40635 } }
I20260927 04:07:15.243736 7291 leader_election.cc:302] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bfedf1c8a2747a3b8fd21758c541896; no voters:
I20260927 04:07:15.243945 7291 leader_election.cc:288] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:15.243968 7296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:15.244086 7296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [term 1 LEADER]: Becoming Leader. State: Replica: 6bfedf1c8a2747a3b8fd21758c541896, State: Running, Role: LEADER
I20260927 04:07:15.244253 7291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:15.244238 7296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfedf1c8a2747a3b8fd21758c541896" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40635 } }
I20260927 04:07:15.244717 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bfedf1c8a2747a3b8fd21758c541896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfedf1c8a2747a3b8fd21758c541896" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40635 } } }
I20260927 04:07:15.244964 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:15.244886 7297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bfedf1c8a2747a3b8fd21758c541896. Latest consensus state: current_term: 1 leader_uuid: "6bfedf1c8a2747a3b8fd21758c541896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bfedf1c8a2747a3b8fd21758c541896" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40635 } } }
I20260927 04:07:15.245249 7297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:15.245429 7305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 04:07:15 dist-test-slave-zqrn krb5kdc[7210](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:07:15.246053 7305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:15.247846 7305 catalog_manager.cc:1347] Generated new cluster ID: 73a52b15fffa465fa727ffde68073f80
I20260927 04:07:15.247901 7305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:15.250156 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:15.239142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37367 (local address 127.17.184.254:40635)
0927 04:07:15.239323 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:15.239332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:15.239364 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:07:15.239877 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:15.239891 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:15.239941 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:15.240097 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:15.240777 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:15.241686 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:15.242511 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:15.243113 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:15.243242 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:15.246300 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:15.246305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:15.246305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:15.246319 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:15.248562 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:15.248993 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:15.248996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:15.248996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:15.249104 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:15.249342 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:15.249343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:15.249344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:15.249589 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:15.249782 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:15.249878 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 04:07:15.249981 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260927 04:07:15.252205 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:15.261325 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:15.253563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45747 (local address 127.17.184.254:40635)
0927 04:07:15.253603 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:15.253605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:15.255083 (+ 1478us) server_negotiation.cc:407] Connection header received
0927 04:07:15.255175 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:15.255176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:15.255185 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:15.255252 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:15.255796 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:15.256304 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:15.257075 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:15.257294 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:15.257385 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:15.258634 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:15.258635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:15.258636 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:15.258642 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:15.260098 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:15.260470 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:15.260471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:15.260471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:15.260533 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:15.260761 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:15.260762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:15.260762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:15.260876 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:15.260992 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:15.261139 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 04:07:15.261203 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:07:15.261917 7305 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:15.262594 7305 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:15.269620 7305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bfedf1c8a2747a3b8fd21758c541896: Generated new TSK 0
I20260927 04:07:15.269865 7305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:15.276643 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7223
W20260927 04:07:15.280961 7208 connection.cc:535] client connection to 127.17.184.254:40635 recv error: Network error: failed to read from TLS socket (remote: 127.17.184.254:40635): Connection reset by peer (error 104)
2026-09-27T04:07:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:15 dist-test-slave-zqrn krb5kdc[7322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:15 dist-test-slave-zqrn krb5kdc[7322](info): set up 2 sockets
Sep 27 04:07:15 dist-test-slave-zqrn krb5kdc[7322](info): commencing operation
krb5kdc: starting...
W20260927 04:07:16.311213 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:16 dist-test-slave-zqrn krb5kdc[7322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:07:16.445385 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:40767
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:40767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:16.528267 7335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:16.528548 7335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:16.528592 7335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:16.529785 7335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:16.529871 7335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:16.529896 7335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:16.529917 7335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:16.529944 7335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:16.531615 7335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:40767
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:16.531929 7335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:16.532207 7335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:16.537443 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:16.537443 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:16.537542 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:16.537681 7335 server_base.cc:1034] running on GCE node
I20260927 04:07:16.537958 7335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:16.538249 7335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:16.539429 7335 hybrid_clock.cc:648] HybridClock initialized: now 1790482036539392 us; error 51 us; skew 500 ppm
Sep 27 04:07:16 dist-test-slave-zqrn krb5kdc[7322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482036, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:16.542229 7335 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:07:16.542742 7335 webserver.cc:460] Webserver started at http://127.17.184.254:40047/ using document root <none> and password file <none>
I20260927 04:07:16.543047 7335 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:16.543118 7335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:16.543264 7335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:16.544373 7335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "2ef8f4bf3bf94074a8c6161309ad145a"
format_stamp: "Formatted at 2026-09-27 04:07:16 on dist-test-slave-zqrn"
server_key: "8b3ebb092d06a800b71d1ed3c411d3aa"
server_key_iv: "f6f835826aefd1e23a6f2c0b78cd76cb"
server_key_version: "encryptionkey@0"
I20260927 04:07:16.544766 7335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "2ef8f4bf3bf94074a8c6161309ad145a"
format_stamp: "Formatted at 2026-09-27 04:07:16 on dist-test-slave-zqrn"
server_key: "8b3ebb092d06a800b71d1ed3c411d3aa"
server_key_iv: "f6f835826aefd1e23a6f2c0b78cd76cb"
server_key_version: "encryptionkey@0"
I20260927 04:07:16.546196 7335 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:07:16.547168 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:16.547406 7335 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:16.547513 7335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "2ef8f4bf3bf94074a8c6161309ad145a"
format_stamp: "Formatted at 2026-09-27 04:07:16 on dist-test-slave-zqrn"
server_key: "8b3ebb092d06a800b71d1ed3c411d3aa"
server_key_iv: "f6f835826aefd1e23a6f2c0b78cd76cb"
server_key_version: "encryptionkey@0"
I20260927 04:07:16.547610 7335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:16.565654 7335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:16.567826 7335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:16.568022 7335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:16.572706 7335 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:40767
I20260927 04:07:16.572744 7402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:40767 every 8 connection(s)
I20260927 04:07:16.573119 7335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:16.573993 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:16.576308 7403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a: Bootstrap starting.
I20260927 04:07:16.576835 7403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:16.577085 7403 log.cc:824] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:16.577726 7403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a: No bootstrap required, opened a new log
I20260927 04:07:16.579303 7403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef8f4bf3bf94074a8c6161309ad145a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40767 } }
I20260927 04:07:16.579443 7403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:16.579468 7403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8f4bf3bf94074a8c6161309ad145a, State: Initialized, Role: FOLLOWER
I20260927 04:07:16.579576 7403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef8f4bf3bf94074a8c6161309ad145a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40767 } }
I20260927 04:07:16.579676 7403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:16.579715 7403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:16.579766 7403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:16.580644 7403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef8f4bf3bf94074a8c6161309ad145a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40767 } }
I20260927 04:07:16.580777 7403 leader_election.cc:302] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ef8f4bf3bf94074a8c6161309ad145a; no voters:
I20260927 04:07:16.580965 7403 leader_election.cc:288] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:16.580988 7406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:16.580981 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7335
I20260927 04:07:16.581099 7406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [term 1 LEADER]: Becoming Leader. State: Replica: 2ef8f4bf3bf94074a8c6161309ad145a, State: Running, Role: LEADER
I20260927 04:07:16.581157 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:16.581244 7406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef8f4bf3bf94074a8c6161309ad145a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40767 } }
I20260927 04:07:16.581357 18147 external_mini_cluster.cc:1428] Setting key a1149123072c822a9d3734f9ee3bf980
I20260927 04:07:16.581249 7403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:16.581724 7407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ef8f4bf3bf94074a8c6161309ad145a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef8f4bf3bf94074a8c6161309ad145a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40767 } } }
I20260927 04:07:16.581820 7407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:16.582068 7408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ef8f4bf3bf94074a8c6161309ad145a. Latest consensus state: current_term: 1 leader_uuid: "2ef8f4bf3bf94074a8c6161309ad145a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ef8f4bf3bf94074a8c6161309ad145a" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 40767 } } }
I20260927 04:07:16.582211 7408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:16.582437 7413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:16.583065 7413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:16.584905 7413 catalog_manager.cc:1347] Generated new cluster ID: 3837054c8e794f9eb543988224a3eabb
I20260927 04:07:16.584959 7413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:16.587800 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:16.582040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46587 (local address 127.17.184.254:40767)
0927 04:07:16.582566 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:16.582588 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:16.582629 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:07:16.582767 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:16.582776 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:16.582846 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:16.583063 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:16.583290 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:16.584873 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:16.585754 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:16.586190 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:16.586337 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:16.587150 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:16.587154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:16.587154 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:16.587338 (+ 184us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:16.587368 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:16.587379 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:16.587478 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:16.587557 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:07:16.587595 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":74,"threads_started":1}
I20260927 04:07:16.588744 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:16.593413 7417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:16.590162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41239 (local address 127.17.184.254:40767)
0927 04:07:16.590211 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:16.590213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:16.590345 (+ 132us) server_negotiation.cc:407] Connection header received
0927 04:07:16.590419 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:16.590421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:16.590431 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:16.590486 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:16.590659 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:16.591347 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:16.592099 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:16.592365 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:16.592448 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:16.593146 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:16.593148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:16.593148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:16.593154 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:16.593168 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:16.593172 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:16.593226 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:16.593286 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 04:07:16.593311 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260927 04:07:16.607753 7413 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:16.608498 7413 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:16.621248 7413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ef8f4bf3bf94074a8c6161309ad145a: Generated new TSK 0
I20260927 04:07:16.621480 7413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:16.627107 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7335
W20260927 04:07:16.632360 7320 connection.cc:535] client connection to 127.17.184.254:40767 recv error: Network error: failed to read from TLS socket (remote: 127.17.184.254:40767): Connection reset by peer (error 104)
2026-09-27T04:07:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:16 dist-test-slave-zqrn krb5kdc[7434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:16 dist-test-slave-zqrn krb5kdc[7434](info): set up 2 sockets
Sep 27 04:07:16 dist-test-slave-zqrn krb5kdc[7434](info): commencing operation
krb5kdc: starting...
W20260927 04:07:17.671252 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:17 dist-test-slave-zqrn krb5kdc[7434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:07:17.794795 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:39439
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:39439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:17.877291 7447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:17.877555 7447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:17.877593 7447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:17.878818 7447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:17.878899 7447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:17.878947 7447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:17.879002 7447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:17.879032 7447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:17.880576 7447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:39439
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:39439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:17.880890 7447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:17.881138 7447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:17.886332 7455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:17.886495 7447 server_base.cc:1034] running on GCE node
W20260927 04:07:17.886353 7453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:17.886397 7452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:17.886883 7447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:17.887256 7447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:17.888437 7447 hybrid_clock.cc:648] HybridClock initialized: now 1790482037888415 us; error 32 us; skew 500 ppm
Sep 27 04:07:17 dist-test-slave-zqrn krb5kdc[7434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482037, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:17.891141 7447 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:07:17.891636 7447 webserver.cc:460] Webserver started at http://127.17.184.254:40293/ using document root <none> and password file <none>
I20260927 04:07:17.891886 7447 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:17.891933 7447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:17.892073 7447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:17.893193 7447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "20f2d79a977e4dc49fcfc613074434bf"
format_stamp: "Formatted at 2026-09-27 04:07:17 on dist-test-slave-zqrn"
server_key: "eeb009fd3861e97cebbb0563a3630449"
server_key_iv: "f9ba053ce9e1ef5f8015fcb6072ec4fb"
server_key_version: "encryptionkey@0"
I20260927 04:07:17.893567 7447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "20f2d79a977e4dc49fcfc613074434bf"
format_stamp: "Formatted at 2026-09-27 04:07:17 on dist-test-slave-zqrn"
server_key: "eeb009fd3861e97cebbb0563a3630449"
server_key_iv: "f9ba053ce9e1ef5f8015fcb6072ec4fb"
server_key_version: "encryptionkey@0"
I20260927 04:07:17.895027 7447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:07:17.895864 7462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:17.896046 7447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:07:17.896183 7447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "20f2d79a977e4dc49fcfc613074434bf"
format_stamp: "Formatted at 2026-09-27 04:07:17 on dist-test-slave-zqrn"
server_key: "eeb009fd3861e97cebbb0563a3630449"
server_key_iv: "f9ba053ce9e1ef5f8015fcb6072ec4fb"
server_key_version: "encryptionkey@0"
I20260927 04:07:17.896268 7447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:17.899721 7447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:17.902227 7447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:17.902390 7447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:17.906646 7447 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:39439
I20260927 04:07:17.906675 7514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:39439 every 8 connection(s)
I20260927 04:07:17.907109 7447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:17.908118 7515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:17.909361 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7447
I20260927 04:07:17.909494 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:17.909695 18147 external_mini_cluster.cc:1428] Setting key c49a23d7124bc356c1912f4989492e63
I20260927 04:07:17.910697 7515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf: Bootstrap starting.
I20260927 04:07:17.911327 7515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:17.911624 7515 log.cc:824] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:17.912351 7515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf: No bootstrap required, opened a new log
I20260927 04:07:17.912406 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:17.910231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38005 (local address 127.17.184.254:39439)
0927 04:07:17.910439 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:17.910448 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:17.910479 (+ 31us) server_negotiation.cc:407] Connection header received
0927 04:07:17.911011 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:17.911018 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:17.911046 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:17.911220 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:17.911988 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:17.911992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:17.911992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:17.911996 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:17.912042 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:17.912053 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:17.912149 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:17.912200 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 04:07:17.912242 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260927 04:07:17.913467 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:17.914686 7515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f2d79a977e4dc49fcfc613074434bf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39439 } }
I20260927 04:07:17.914932 7515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:17.915001 7515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20f2d79a977e4dc49fcfc613074434bf, State: Initialized, Role: FOLLOWER
I20260927 04:07:17.915172 7515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f2d79a977e4dc49fcfc613074434bf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39439 } }
I20260927 04:07:17.915273 7515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:17.915298 7515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:17.915380 7515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:17.916015 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:17.914828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54473 (local address 127.17.184.254:39439)
0927 04:07:17.914862 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:17.914864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:17.915008 (+ 144us) server_negotiation.cc:407] Connection header received
0927 04:07:17.915066 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:17.915067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:17.915073 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:17.915139 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:17.915796 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:17.915797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:17.915798 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:17.915803 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:17.915817 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:17.915822 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:17.915891 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:17.915926 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 04:07:17.915939 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 04:07:17.916040 7515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f2d79a977e4dc49fcfc613074434bf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39439 } }
I20260927 04:07:17.916142 7515 leader_election.cc:302] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20f2d79a977e4dc49fcfc613074434bf; no voters:
I20260927 04:07:17.916299 7515 leader_election.cc:288] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:17.916326 7525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:17.916432 7525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [term 1 LEADER]: Becoming Leader. State: Replica: 20f2d79a977e4dc49fcfc613074434bf, State: Running, Role: LEADER
I20260927 04:07:17.916548 7515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:17.916565 7525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f2d79a977e4dc49fcfc613074434bf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39439 } }
I20260927 04:07:17.917016 7526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20f2d79a977e4dc49fcfc613074434bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f2d79a977e4dc49fcfc613074434bf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39439 } } }
I20260927 04:07:17.917033 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20f2d79a977e4dc49fcfc613074434bf. Latest consensus state: current_term: 1 leader_uuid: "20f2d79a977e4dc49fcfc613074434bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20f2d79a977e4dc49fcfc613074434bf" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 39439 } } }
I20260927 04:07:17.917198 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:17.917433 7534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:17.917588 7526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:17.918217 7534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:17.919862 7534 catalog_manager.cc:1347] Generated new cluster ID: 1fd23bc21ba941a3afe6bcded3bbdd75
I20260927 04:07:17.919905 7534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:17.935330 7534 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:17.935562 7534 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:17.940661 7534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20f2d79a977e4dc49fcfc613074434bf: Generated new TSK 0
I20260927 04:07:17.940898 7534 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:17.947399 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7447
2026-09-27T04:07:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:18 dist-test-slave-zqrn krb5kdc[7546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:18 dist-test-slave-zqrn krb5kdc[7546](info): set up 2 sockets
Sep 27 04:07:18 dist-test-slave-zqrn krb5kdc[7546](info): commencing operation
krb5kdc: starting...
W20260927 04:07:18.987201 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:19 dist-test-slave-zqrn krb5kdc[7546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:07:19.113984 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44755
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:44755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:19.199059 7559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:19.199271 7559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:19.199294 7559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:19.200404 7559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:19.200439 7559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:19.200454 7559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:19.200466 7559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:19.200479 7559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:19.201864 7559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:44755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:44755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:19.202049 7559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:19.202211 7559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 04:07:19.207363 7559 server_base.cc:1034] running on GCE node
W20260927 04:07:19.207335 7565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:19.207327 7564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:19.207432 7567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:19.207770 7559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:19.208057 7559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:19.209223 7559 hybrid_clock.cc:648] HybridClock initialized: now 1790482039209216 us; error 37 us; skew 500 ppm
Sep 27 04:07:19 dist-test-slave-zqrn krb5kdc[7546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482039, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:19.211769 7559 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:07:19.212239 7559 webserver.cc:460] Webserver started at http://127.17.184.254:46461/ using document root <none> and password file <none>
I20260927 04:07:19.212420 7559 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:19.212462 7559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:19.212550 7559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:19.213538 7559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "89fe7b1e88ac464f9a39902145f016a9"
format_stamp: "Formatted at 2026-09-27 04:07:19 on dist-test-slave-zqrn"
server_key: "a041d86766af442ec199a776b0a14504"
server_key_iv: "2b8d0ca31b649ba3f760ea3bf56e0b9c"
server_key_version: "encryptionkey@0"
I20260927 04:07:19.213843 7559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "89fe7b1e88ac464f9a39902145f016a9"
format_stamp: "Formatted at 2026-09-27 04:07:19 on dist-test-slave-zqrn"
server_key: "a041d86766af442ec199a776b0a14504"
server_key_iv: "2b8d0ca31b649ba3f760ea3bf56e0b9c"
server_key_version: "encryptionkey@0"
I20260927 04:07:19.215173 7559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:07:19.215852 7574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:19.216028 7559 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260927 04:07:19.216096 7559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "89fe7b1e88ac464f9a39902145f016a9"
format_stamp: "Formatted at 2026-09-27 04:07:19 on dist-test-slave-zqrn"
server_key: "a041d86766af442ec199a776b0a14504"
server_key_iv: "2b8d0ca31b649ba3f760ea3bf56e0b9c"
server_key_version: "encryptionkey@0"
I20260927 04:07:19.216171 7559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:19.223717 7559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:19.225917 7559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:19.226028 7559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:19.230146 7559 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:44755
I20260927 04:07:19.230187 7626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:44755 every 8 connection(s)
I20260927 04:07:19.230556 7559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:19.231554 7627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:19.234038 7627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9: Bootstrap starting.
I20260927 04:07:19.234663 7627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:19.235026 7627 log.cc:824] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:19.235824 7627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9: No bootstrap required, opened a new log
I20260927 04:07:19.237480 7627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fe7b1e88ac464f9a39902145f016a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44755 } }
I20260927 04:07:19.237617 7627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:19.237663 7627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89fe7b1e88ac464f9a39902145f016a9, State: Initialized, Role: FOLLOWER
I20260927 04:07:19.237812 7627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fe7b1e88ac464f9a39902145f016a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44755 } }
I20260927 04:07:19.237874 7627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:19.237895 7627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:19.237939 7627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:19.238833 7627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fe7b1e88ac464f9a39902145f016a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44755 } }
I20260927 04:07:19.238859 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7559
I20260927 04:07:19.238999 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:19.238971 7627 leader_election.cc:302] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89fe7b1e88ac464f9a39902145f016a9; no voters:
I20260927 04:07:19.239145 7627 leader_election.cc:288] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:19.239203 7630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:19.239233 18147 external_mini_cluster.cc:1428] Setting key 8a6bf24d4c856e04ebb38d5c9a8b6f2e
I20260927 04:07:19.239364 7630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [term 1 LEADER]: Becoming Leader. State: Replica: 89fe7b1e88ac464f9a39902145f016a9, State: Running, Role: LEADER
I20260927 04:07:19.239408 7627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:19.239501 7630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fe7b1e88ac464f9a39902145f016a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44755 } }
I20260927 04:07:19.239928 7631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89fe7b1e88ac464f9a39902145f016a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fe7b1e88ac464f9a39902145f016a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44755 } } }
I20260927 04:07:19.239928 7632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89fe7b1e88ac464f9a39902145f016a9. Latest consensus state: current_term: 1 leader_uuid: "89fe7b1e88ac464f9a39902145f016a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89fe7b1e88ac464f9a39902145f016a9" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 44755 } } }
I20260927 04:07:19.240034 7631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:19.240041 7632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:19.240747 7640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:19.241355 7640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:19.242787 7640 catalog_manager.cc:1347] Generated new cluster ID: b50877d2151b48658c78c660ae4bbb72
I20260927 04:07:19.242837 7640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:07:19 dist-test-slave-zqrn krb5kdc[7546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:07:19.251374 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:19.239892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37709 (local address 127.17.184.254:44755)
0927 04:07:19.240272 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:19.240282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:19.240314 (+ 32us) server_negotiation.cc:407] Connection header received
0927 04:07:19.240431 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:19.240438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:19.240496 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:19.241239 (+ 743us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:19.241531 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:19.242954 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:19.244050 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:19.244701 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:19.244818 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:19.247665 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:19.247669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:19.247670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:19.247685 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:19.249699 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:19.250092 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:19.250096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:19.250096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:19.250215 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:19.250387 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:19.250388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:19.250389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:19.250746 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:19.250937 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:19.251015 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 04:07:19.251156 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":71,"threads_started":1}
I20260927 04:07:19.252235 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:19.257508 7640 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:19.258226 7640 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:19.260881 7635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:19.253733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39035 (local address 127.17.184.254:44755)
0927 04:07:19.253768 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:19.253769 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:19.253776 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:07:19.253799 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:19.253799 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:19.253809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:19.253866 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:19.254461 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:19.255251 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:19.256234 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:19.256565 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:19.256637 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:19.258089 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:19.258090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:19.258091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:19.258098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:19.259770 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:19.260137 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:19.260138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:19.260138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:19.260197 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:19.260385 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:19.260386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:19.260387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:19.260516 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:19.260620 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:19.260708 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 04:07:19.260794 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 04:07:19.266273 7640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89fe7b1e88ac464f9a39902145f016a9: Generated new TSK 0
I20260927 04:07:19.266464 7640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:19.272845 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7559
2026-09-27T04:07:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:19 dist-test-slave-zqrn krb5kdc[7658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:19 dist-test-slave-zqrn krb5kdc[7658](info): set up 2 sockets
Sep 27 04:07:19 dist-test-slave-zqrn krb5kdc[7658](info): commencing operation
krb5kdc: starting...
W20260927 04:07:20.323606 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:20 dist-test-slave-zqrn krb5kdc[7658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:07:20.446511 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:38255
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:38255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:20.527311 7671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:20.527555 7671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:20.527593 7671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:20.528755 7671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:20.528824 7671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:20.528849 7671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:20.528877 7671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:20.528909 7671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:20.530385 7671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:38255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:38255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:20.530673 7671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:20.530931 7671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:20.535807 7676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:20.535818 7679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:20.535835 7677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:20.536059 7671 server_base.cc:1034] running on GCE node
I20260927 04:07:20.536336 7671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:20.536624 7671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:20.537810 7671 hybrid_clock.cc:648] HybridClock initialized: now 1790482040537805 us; error 38 us; skew 500 ppm
Sep 27 04:07:20 dist-test-slave-zqrn krb5kdc[7658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482040, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:20.546344 7671 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:07:20.546952 7671 webserver.cc:460] Webserver started at http://127.17.184.254:33541/ using document root <none> and password file <none>
I20260927 04:07:20.547196 7671 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:20.547264 7671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:20.547415 7671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:20.548506 7671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "aca3976079a34c8bad72e6c2228f6eb6"
format_stamp: "Formatted at 2026-09-27 04:07:20 on dist-test-slave-zqrn"
server_key: "ee907e53ab059ceb3def427f2f2076cc"
server_key_iv: "1e24eb3fd4badf9fce2538da39d7a583"
server_key_version: "encryptionkey@0"
I20260927 04:07:20.548863 7671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "aca3976079a34c8bad72e6c2228f6eb6"
format_stamp: "Formatted at 2026-09-27 04:07:20 on dist-test-slave-zqrn"
server_key: "ee907e53ab059ceb3def427f2f2076cc"
server_key_iv: "1e24eb3fd4badf9fce2538da39d7a583"
server_key_version: "encryptionkey@0"
I20260927 04:07:20.550237 7671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:07:20.551056 7686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:20.551240 7671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:20.551342 7671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "aca3976079a34c8bad72e6c2228f6eb6"
format_stamp: "Formatted at 2026-09-27 04:07:20 on dist-test-slave-zqrn"
server_key: "ee907e53ab059ceb3def427f2f2076cc"
server_key_iv: "1e24eb3fd4badf9fce2538da39d7a583"
server_key_version: "encryptionkey@0"
I20260927 04:07:20.551424 7671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:20.562674 7671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:20.564707 7671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:20.564873 7671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:20.568908 7671 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:38255
I20260927 04:07:20.568941 7738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:38255 every 8 connection(s)
I20260927 04:07:20.569298 7671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:20.570178 7739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:20.571798 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7671
I20260927 04:07:20.571925 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:20.572127 18147 external_mini_cluster.cc:1428] Setting key c4ba5479812fb6c117c56855050a5ce6
I20260927 04:07:20.572995 7739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6: Bootstrap starting.
I20260927 04:07:20.573648 7739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:20.573956 7739 log.cc:824] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:20.574769 7739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6: No bootstrap required, opened a new log
I20260927 04:07:20.577221 7739 raft_consensus.cc:348] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3976079a34c8bad72e6c2228f6eb6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38255 } }
I20260927 04:07:20.577404 7739 raft_consensus.cc:374] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:20.577464 7739 raft_consensus.cc:729] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca3976079a34c8bad72e6c2228f6eb6, State: Initialized, Role: FOLLOWER
I20260927 04:07:20.577595 7739 consensus_queue.cc:260] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3976079a34c8bad72e6c2228f6eb6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38255 } }
I20260927 04:07:20.577702 7739 raft_consensus.cc:388] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:20.577744 7739 raft_consensus.cc:482] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:20.577790 7739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:20.578235 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:20.572675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58635 (local address 127.17.184.254:38255)
0927 04:07:20.572873 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:20.572881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:20.572907 (+ 26us) server_negotiation.cc:407] Connection header received
0927 04:07:20.573367 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:20.573374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:20.573421 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:20.573585 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:20.573838 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:20.575030 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:20.576152 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:20.576583 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:20.576675 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:20.577396 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:20.577400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:20.577400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:20.577696 (+ 296us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:20.577754 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:20.577769 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:20.577843 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:20.577921 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 04:07:20.578007 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260927 04:07:20.578433 7739 raft_consensus.cc:504] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3976079a34c8bad72e6c2228f6eb6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38255 } }
I20260927 04:07:20.578549 7739 leader_election.cc:302] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aca3976079a34c8bad72e6c2228f6eb6; no voters:
I20260927 04:07:20.578701 7739 leader_election.cc:288] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:20.578739 7744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:20.578876 7744 raft_consensus.cc:686] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [term 1 LEADER]: Becoming Leader. State: Replica: aca3976079a34c8bad72e6c2228f6eb6, State: Running, Role: LEADER
I20260927 04:07:20.578997 7739 sys_catalog.cc:564] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:20.578984 7744 consensus_queue.cc:237] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3976079a34c8bad72e6c2228f6eb6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38255 } }
I20260927 04:07:20.579181 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:20.579308 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aca3976079a34c8bad72e6c2228f6eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3976079a34c8bad72e6c2228f6eb6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38255 } } }
I20260927 04:07:20.579380 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:20.579300 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aca3976079a34c8bad72e6c2228f6eb6. Latest consensus state: current_term: 1 leader_uuid: "aca3976079a34c8bad72e6c2228f6eb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aca3976079a34c8bad72e6c2228f6eb6" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 38255 } } }
I20260927 04:07:20.579538 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6 [sys.catalog]: This master's current role is: LEADER
W20260927 04:07:20.580856 7762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:20.580955 7762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:20.581043 7749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:20.581269 7749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:20.582844 7749 catalog_manager.cc:1347] Generated new cluster ID: de10eb26559949e28573230d4e863530
I20260927 04:07:20.582924 7749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:20.584244 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:20.580817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52211 (local address 127.17.184.254:38255)
0927 04:07:20.580853 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:20.580855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:20.580960 (+ 105us) server_negotiation.cc:407] Connection header received
0927 04:07:20.581067 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:20.581068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:20.581078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:20.581108 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:20.581317 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:20.581980 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:20.582949 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:20.583235 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:20.583275 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:20.584004 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:20.584005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:20.584006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:20.584010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:20.584031 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:20.584035 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:20.584061 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:20.584155 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 04:07:20.584175 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:07:20.587922 7749 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:20.588640 7749 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:20.596256 7749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aca3976079a34c8bad72e6c2228f6eb6: Generated new TSK 0
I20260927 04:07:20.596493 7749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:20.606626 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7671
2026-09-27T04:07:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:20 dist-test-slave-zqrn krb5kdc[7770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:20 dist-test-slave-zqrn krb5kdc[7770](info): set up 2 sockets
Sep 27 04:07:20 dist-test-slave-zqrn krb5kdc[7770](info): commencing operation
krb5kdc: starting...
W20260927 04:07:21.651257 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:21 dist-test-slave-zqrn krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:07:21.777556 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33605
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:33605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:21.857865 7783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:21.858088 7783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:21.858112 7783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:21.859259 7783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:21.859297 7783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:21.859311 7783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:21.859324 7783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:21.859336 7783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:21.860776 7783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:33605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:21.860975 7783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:21.861162 7783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:21.866180 7788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:21.866250 7783 server_base.cc:1034] running on GCE node
W20260927 04:07:21.866214 7789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:21.866185 7791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:21.866603 7783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:21.866941 7783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:21.868110 7783 hybrid_clock.cc:648] HybridClock initialized: now 1790482041868098 us; error 31 us; skew 500 ppm
Sep 27 04:07:21 dist-test-slave-zqrn krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482041, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:21.870675 7783 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:07:21.871143 7783 webserver.cc:460] Webserver started at http://127.17.184.254:40035/ using document root <none> and password file <none>
I20260927 04:07:21.871380 7783 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:21.871425 7783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:21.871600 7783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:21.872711 7783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "15ca596ebe644e4c8f6793eb7a597221"
format_stamp: "Formatted at 2026-09-27 04:07:21 on dist-test-slave-zqrn"
server_key: "d39a686c2fd5edfe4b3ba02c55c56dcf"
server_key_iv: "173e22f569b00262a7aa6fe92b473f09"
server_key_version: "encryptionkey@0"
I20260927 04:07:21.873064 7783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "15ca596ebe644e4c8f6793eb7a597221"
format_stamp: "Formatted at 2026-09-27 04:07:21 on dist-test-slave-zqrn"
server_key: "d39a686c2fd5edfe4b3ba02c55c56dcf"
server_key_iv: "173e22f569b00262a7aa6fe92b473f09"
server_key_version: "encryptionkey@0"
I20260927 04:07:21.874392 7783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260927 04:07:21.875192 7798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:21.875393 7783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:21.875487 7783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "15ca596ebe644e4c8f6793eb7a597221"
format_stamp: "Formatted at 2026-09-27 04:07:21 on dist-test-slave-zqrn"
server_key: "d39a686c2fd5edfe4b3ba02c55c56dcf"
server_key_iv: "173e22f569b00262a7aa6fe92b473f09"
server_key_version: "encryptionkey@0"
I20260927 04:07:21.875560 7783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:21.878814 7783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:21.880791 7783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:21.880937 7783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:21.885332 7783 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:33605
I20260927 04:07:21.885397 7850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:33605 every 8 connection(s)
I20260927 04:07:21.885707 7783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:21.886643 7851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:21.888973 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221: Bootstrap starting.
I20260927 04:07:21.889564 7851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:21.889860 7851 log.cc:824] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:21.890558 7851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221: No bootstrap required, opened a new log
I20260927 04:07:21.892237 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7783
I20260927 04:07:21.892372 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:21.892374 7851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ca596ebe644e4c8f6793eb7a597221" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33605 } }
I20260927 04:07:21.892570 7851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 04:07:21.892571 18147 external_mini_cluster.cc:1428] Setting key f9b0424605ffc7d461118a067fef47e5
I20260927 04:07:21.892596 7851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15ca596ebe644e4c8f6793eb7a597221, State: Initialized, Role: FOLLOWER
I20260927 04:07:21.892776 7851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ca596ebe644e4c8f6793eb7a597221" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33605 } }
I20260927 04:07:21.892869 7851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 04:07:21.892905 7851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 04:07:21.892963 7851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:21.894171 7851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ca596ebe644e4c8f6793eb7a597221" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33605 } }
I20260927 04:07:21.894297 7851 leader_election.cc:302] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15ca596ebe644e4c8f6793eb7a597221; no voters:
I20260927 04:07:21.894474 7851 leader_election.cc:288] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 04:07:21.894587 7856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:21.894781 7856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [term 1 LEADER]: Becoming Leader. State: Replica: 15ca596ebe644e4c8f6793eb7a597221, State: Running, Role: LEADER
I20260927 04:07:21.894821 7851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:21.894972 7856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ca596ebe644e4c8f6793eb7a597221" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33605 } }
I20260927 04:07:21.895015 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:21.893137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35835 (local address 127.17.184.254:33605)
0927 04:07:21.893346 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:21.893356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:21.893394 (+ 38us) server_negotiation.cc:407] Connection header received
0927 04:07:21.893527 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:21.893537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:21.893568 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:21.893770 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:21.894389 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:21.894393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:21.894394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:21.894399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:21.894448 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:21.894464 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:21.894542 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:21.894621 (+ 79us) server_negotiation.cc:299] Negotiation successful
0927 04:07:21.894767 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260927 04:07:21.895354 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15ca596ebe644e4c8f6793eb7a597221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ca596ebe644e4c8f6793eb7a597221" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33605 } } }
I20260927 04:07:21.895429 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:21.895701 7862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:21.895994 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:21.896246 7862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:21.897781 7858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15ca596ebe644e4c8f6793eb7a597221. Latest consensus state: current_term: 1 leader_uuid: "15ca596ebe644e4c8f6793eb7a597221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15ca596ebe644e4c8f6793eb7a597221" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 33605 } } }
I20260927 04:07:21.897866 7858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:21.897998 7862 catalog_manager.cc:1347] Generated new cluster ID: c813634cf0c542b9930527ec0234e098
I20260927 04:07:21.898061 7862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:21.899192 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:21.897648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50991 (local address 127.17.184.254:33605)
0927 04:07:21.897821 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:21.897822 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:21.898174 (+ 352us) server_negotiation.cc:407] Connection header received
0927 04:07:21.898194 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:21.898195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:21.898199 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:21.898259 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:21.898931 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:21.898932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:21.898933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 04:07:21.898937 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:21.898951 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 04:07:21.898956 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:21.899039 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:21.899099 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 04:07:21.899116 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260927 04:07:21.909142 7862 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:21.909327 7862 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:21.916009 7862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15ca596ebe644e4c8f6793eb7a597221: Generated new TSK 0
I20260927 04:07:21.916199 7862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:21.918505 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7783
2026-09-27T04:07:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1312 ms)
[----------] 18 tests from AuthTokenIssuingTest (24262 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:21 dist-test-slave-zqrn krb5kdc[7882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:21 dist-test-slave-zqrn krb5kdc[7882](info): set up 2 sockets
Sep 27 04:07:21 dist-test-slave-zqrn krb5kdc[7882](info): commencing operation
krb5kdc: starting...
W20260927 04:07:22.995249 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:07:23.126679 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:42633
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:42633,127.17.184.253:44199,127.17.184.252:44333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:23.207563 7895 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:07:23.207770 7895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:23.207788 7895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:23.207808 7895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:23.208894 7895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:23.208927 7895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:23.208941 7895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:23.208954 7895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:23.208966 7895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:23.210354 7895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:42633,127.17.184.253:44199,127.17.184.252:44333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:42633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:23.210552 7895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:23.210714 7895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:23.215557 7900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:23.215675 7903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:23.215615 7901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:23.215932 7895 server_base.cc:1034] running on GCE node
I20260927 04:07:23.216187 7895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:23.216526 7895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:23.217696 7895 hybrid_clock.cc:648] HybridClock initialized: now 1790482043217684 us; error 32 us; skew 500 ppm
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:23.220492 7895 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:07:23.221007 7895 webserver.cc:460] Webserver started at http://127.17.184.254:46783/ using document root <none> and password file <none>
I20260927 04:07:23.221253 7895 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:23.221325 7895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:23.221484 7895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:23.222661 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "70d0945c09a54be1a8523789f804be25"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "2b1b5c8538dc9dd40daad93cebadaaaa"
server_key_iv: "51b156840e3c3eba3a2b03eaff03ca0d"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.223044 7895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "70d0945c09a54be1a8523789f804be25"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "2b1b5c8538dc9dd40daad93cebadaaaa"
server_key_iv: "51b156840e3c3eba3a2b03eaff03ca0d"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.224481 7895 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:07:23.225454 7910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:23.225725 7895 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:07:23.225837 7895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "70d0945c09a54be1a8523789f804be25"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "2b1b5c8538dc9dd40daad93cebadaaaa"
server_key_iv: "51b156840e3c3eba3a2b03eaff03ca0d"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.225926 7895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:23.240967 7895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:23.243090 7895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:23.243261 7895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:23.247284 7895 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:42633
I20260927 04:07:23.247319 7962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:42633 every 8 connection(s)
I20260927 04:07:23.247677 7895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:23.248550 7963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:23.250159 7963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:23.252317 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7895
I20260927 04:07:23.252431 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:23.252619 18147 external_mini_cluster.cc:1428] Setting key 013176af12f6b7fe2780f316c1878080
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.253@KRBTEST.COM; defaulting to no policy
I20260927 04:07:23.262192 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.251067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47702 (local address 127.17.184.254:42633)
0927 04:07:23.251209 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.251214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.251255 (+ 41us) server_negotiation.cc:407] Connection header received
0927 04:07:23.251746 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.251748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.251766 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.251862 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.252643 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.253626 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.254525 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.254877 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.256303 (+ 1426us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.258752 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.258759 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.258759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.258773 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.260542 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.260945 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.260949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.260949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.261032 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.261313 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.261314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.261315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.261612 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.261690 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.261877 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.261995 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
I20260927 04:07:23.262210 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.250857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42633 (local address 127.0.0.1:47702)
0927 04:07:23.251043 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.251081 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.251538 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.251884 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.251887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.252263 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.252511 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.252518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.253743 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.253744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.254421 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.254423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.256307 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.256308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.256403 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.256947 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.256957 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.258624 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.260642 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.260649 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.260651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.260848 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.261113 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.261114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.261114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.261238 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.261685 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.261692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.261756 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.261834 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.262060 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":104,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.253@KRBTEST.COM" created.
I20260927 04:07:23.263092 7963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:23.263454 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.263347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48006)
0927 04:07:23.263396 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.263436 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 04:07:23.263628 7913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
I20260927 04:07:23.264061 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.263963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48020)
0927 04:07:23.264008 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.264044 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 04:07:23.264155 7963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:44199: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab.
Entry for principal kudu/127.17.184.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab.
I20260927 04:07:23.278970 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.253:44199
--webserver_interface=127.17.184.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab
--principal=kudu/127.17.184.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:42633,127.17.184.253:44199,127.17.184.252:44333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:07:23.313400 7963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } attempt: 1
I20260927 04:07:23.313966 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.313819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48032)
0927 04:07:23.313890 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.313935 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 04:07:23.314369 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.314284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48044)
0927 04:07:23.314325 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.314350 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 04:07:23.314487 7963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:44199: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
W20260927 04:07:23.362437 7970 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:07:23.362723 7970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:23.362751 7970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:23.362797 7970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:23.364059 7970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:23.364135 7970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:23.364161 7970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:23.364192 7970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:23.364218 7970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:23.365688 7970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:42633,127.17.184.253:44199,127.17.184.252:44333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab
--principal=kudu/127.17.184.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.253:44199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.184.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:23.365962 7970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:23.366192 7970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:23.371376 7976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:23.371591 7978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:23.371556 7970 server_base.cc:1034] running on GCE node
W20260927 04:07:23.371382 7975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:23.372573 7970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:23.372946 7970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:23.374133 7970 hybrid_clock.cc:648] HybridClock initialized: now 1790482043374013 us; error 131 us; skew 500 ppm
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:23.381352 7970 init.cc:377] Logged in from keytab as kudu/127.17.184.253@KRBTEST.COM (short username kudu)
I20260927 04:07:23.382007 7970 webserver.cc:460] Webserver started at http://127.17.184.253:36203/ using document root <none> and password file <none>
I20260927 04:07:23.382302 7970 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:23.382376 7970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:23.382520 7970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:23.382664 7963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } attempt: 2
I20260927 04:07:23.383121 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.382978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48046)
0927 04:07:23.383049 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.383082 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 04:07:23.383673 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.383559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48048)
0927 04:07:23.383616 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.383642 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 04:07:23.383787 7963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:44199: Network error: Client connection negotiation failed: client connection to 127.17.184.253:44199: connect: Connection refused (error 111)
I20260927 04:07:23.384222 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/data/instance:
uuid: "e5b115e3b9e74c0183e1dfe33a991b77"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "dabbd3986e542ff3b3f39f54feeda430"
server_key_iv: "dc83072fc672ea18b96402c94c815716"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.384730 7970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/wal/instance:
uuid: "e5b115e3b9e74c0183e1dfe33a991b77"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "dabbd3986e542ff3b3f39f54feeda430"
server_key_iv: "dc83072fc672ea18b96402c94c815716"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.386819 7970 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260927 04:07:23.387848 7985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:23.388080 7970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:23.388191 7970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/wal
uuid: "e5b115e3b9e74c0183e1dfe33a991b77"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "dabbd3986e542ff3b3f39f54feeda430"
server_key_iv: "dc83072fc672ea18b96402c94c815716"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.388276 7970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:23.423243 7970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:23.425840 7970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:23.425982 7970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:23.430716 7970 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.253:44199
I20260927 04:07:23.430804 8037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.253:44199 every 8 connection(s)
I20260927 04:07:23.431191 7970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
I20260927 04:07:23.432209 8038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:23.433918 8038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:23.434489 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 7970
I20260927 04:07:23.434612 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-1/wal/instance
I20260927 04:07:23.434808 18147 external_mini_cluster.cc:1428] Setting key f091f9b2447e05d999d9b57ed4c78e1a
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.184.252@KRBTEST.COM; defaulting to no policy
I20260927 04:07:23.445688 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.434782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47718 (local address 127.17.184.254:42633)
0927 04:07:23.434839 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.434841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.435123 (+ 282us) server_negotiation.cc:407] Connection header received
0927 04:07:23.435605 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.435607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.435625 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.435713 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.436521 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.437420 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.439043 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.439426 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.439478 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.442433 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.442436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.442436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.442446 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.444230 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.444699 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.444700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.444700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.444754 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.445071 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.445072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.445072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.445211 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.445266 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.445544 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.445615 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 04:07:23.445896 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.434786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42633 (local address 127.0.0.1:47718)
0927 04:07:23.434974 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.435019 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.435505 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.435807 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.435811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.436213 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.436413 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.436417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.437530 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.437531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.438885 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.438889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.439584 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.439585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.439905 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.440522 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.440535 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.442322 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.444336 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.444343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.444345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.444624 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.444826 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.444826 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.444826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.444981 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.445276 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.445281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.445410 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.445528 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.445659 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":85,"threads_started":1}
I20260927 04:07:23.446597 8038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.252@KRBTEST.COM" created.
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:07:23.455448 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.447025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48058 (local address 127.17.184.253:44199)
0927 04:07:23.447203 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.447208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.447238 (+ 30us) server_negotiation.cc:407] Connection header received
0927 04:07:23.447278 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.447281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.447297 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.447386 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.447644 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.448871 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.449905 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.450255 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.450318 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.452412 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.452420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.452421 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.452428 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.454173 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.454545 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.454548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.454548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.454606 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.454805 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.454806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.454806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.455015 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.455108 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.455218 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.455301 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260927 04:07:23.455494 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.446983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48058)
0927 04:07:23.447035 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.447047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.447101 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.447400 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.447401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.447514 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.447564 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.447565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.448960 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.448961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.449817 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.449820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.450340 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.450341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.450425 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.450851 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.450856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.452305 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.454276 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.454277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.454277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.454445 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.454674 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.454675 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.454675 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.454730 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.455084 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.455085 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.455124 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.455190 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.455373 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":92}
I20260927 04:07:23.456408 8038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:23.456846 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.456709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44328)
0927 04:07:23.456761 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.456814 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:07:23.457005 7989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
I20260927 04:07:23.457479 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.457379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44342)
0927 04:07:23.457428 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.457456 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 04:07:23.457628 8038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:44333: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab.
Entry for principal kudu/127.17.184.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab.
I20260927 04:07:23.464041 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.252:44333
--webserver_interface=127.17.184.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab
--principal=kudu/127.17.184.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:42633,127.17.184.253:44199,127.17.184.252:44333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:07:23.475008 7963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } attempt: 3
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:07:23.482852 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.475388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48068)
0927 04:07:23.475447 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.475460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.475518 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.475689 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.475690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.475874 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.475956 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.475958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.476708 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.476709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.477382 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.477384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.477818 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.477819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.477910 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.478406 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.478417 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.479841 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.481691 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.481692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.481693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.481886 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.482132 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.482133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.482133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.482203 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.482492 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.482493 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.482545 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.482639 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.482745 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 04:07:23.482852 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.475447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48068 (local address 127.17.184.253:44199)
0927 04:07:23.475532 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.475535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.475547 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:07:23.475590 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.475591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.475611 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.475683 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.476049 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.476606 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.477474 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.477735 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.477822 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.479937 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.479938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.479939 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.479947 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.481544 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.481989 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.481990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.481991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.482058 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.482274 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.482275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.482275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.482399 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.482498 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.482663 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.482747 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 04:07:23.483490 7963 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:23.483853 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.483740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44352)
0927 04:07:23.483793 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.483828 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 04:07:23.484303 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.484182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44366)
0927 04:07:23.484229 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.484284 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 04:07:23.484417 7963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:44333: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
I20260927 04:07:23.506872 8038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } attempt: 1
I20260927 04:07:23.507519 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.507321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44376)
0927 04:07:23.507392 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.507436 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 04:07:23.507920 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.507825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44382)
0927 04:07:23.507857 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.507885 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260927 04:07:23.508018 8038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:44333: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
I20260927 04:07:23.515582 7963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } attempt: 1
I20260927 04:07:23.516062 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.515910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44392)
0927 04:07:23.516001 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.516031 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 04:07:23.516414 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.516314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44408)
0927 04:07:23.516360 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.516386 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:07:23.516507 7963 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:44333: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
W20260927 04:07:23.547344 8045 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:07:23.547552 8045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:23.547570 8045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:23.547590 8045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:23.548753 8045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:23.548794 8045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:23.548810 8045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:23.548822 8045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:23.548834 8045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:23.550308 8045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:42633,127.17.184.253:44199,127.17.184.252:44333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab
--principal=kudu/127.17.184.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.252:44333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.184.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:23.550550 8045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:23.550802 8045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:23.556017 8053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:23.556020 8050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:23.556018 8051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:23.556354 8045 server_base.cc:1034] running on GCE node
I20260927 04:07:23.556659 8045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:23.556954 8045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:23.558126 8045 hybrid_clock.cc:648] HybridClock initialized: now 1790482043558100 us; error 37 us; skew 500 ppm
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:23.560791 8045 init.cc:377] Logged in from keytab as kudu/127.17.184.252@KRBTEST.COM (short username kudu)
I20260927 04:07:23.561254 8045 webserver.cc:460] Webserver started at http://127.17.184.252:37541/ using document root <none> and password file <none>
I20260927 04:07:23.561502 8045 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:23.561571 8045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:23.561719 8045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:23.562788 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/data/instance:
uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "48e829702b28d7292c8a2f90f31fa0df"
server_key_iv: "1a70243faf3868e85fcba707ee159ad9"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.563227 8045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/wal/instance:
uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "48e829702b28d7292c8a2f90f31fa0df"
server_key_iv: "1a70243faf3868e85fcba707ee159ad9"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.564574 8045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:23.565344 8060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:23.565560 8045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:23.565663 8045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/wal
uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a"
format_stamp: "Formatted at 2026-09-27 04:07:23 on dist-test-slave-zqrn"
server_key: "48e829702b28d7292c8a2f90f31fa0df"
server_key_iv: "1a70243faf3868e85fcba707ee159ad9"
server_key_version: "encryptionkey@0"
I20260927 04:07:23.565739 8045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:23.576177 8038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } attempt: 2
I20260927 04:07:23.576673 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.576532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44422)
0927 04:07:23.576596 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.576640 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 04:07:23.577042 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.576939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44432)
0927 04:07:23.576987 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.577020 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 04:07:23.577131 8038 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:44333: Network error: Client connection negotiation failed: client connection to 127.17.184.252:44333: connect: Connection refused (error 111)
I20260927 04:07:23.578835 8045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:23.581233 8045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:23.581398 8045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:23.585505 8045 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.252:44333
I20260927 04:07:23.585537 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.252:44333 every 8 connection(s)
I20260927 04:07:23.585889 8045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
I20260927 04:07:23.586803 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:23.588414 8113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:23.589097 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 8045
I20260927 04:07:23.589202 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790481773036938-18147-0/minicluster-data/master-2/wal/instance
I20260927 04:07:23.589397 18147 external_mini_cluster.cc:1428] Setting key 62c2035a0102fd0306a005bad9358af5
I20260927 04:07:23.591646 7963 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } attempt: 2
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:07:23.599078 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.589208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.17.184.254:42633)
0927 04:07:23.589262 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.589264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.589519 (+ 255us) server_negotiation.cc:407] Connection header received
0927 04:07:23.590260 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.590262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.590275 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.590330 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.590864 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.591378 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.592694 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.592985 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.593025 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.595671 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.595672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.595673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.595681 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.597312 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.597787 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.597788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.597789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.597875 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.598178 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.598179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.598180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.598373 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.598431 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.598835 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.598951 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 04:07:23.599509 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.589175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:42633 (local address 127.0.0.1:47726)
0927 04:07:23.589337 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.589384 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.590134 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.590346 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.590348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.590609 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.590765 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.590770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.591477 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.591478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.592607 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.592610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.593106 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.593106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.593237 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.593838 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.593848 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.595579 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.597386 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.597393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.597395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.597719 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.597952 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.597953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.597953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.598093 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.598492 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.598498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.598670 (+ 172us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.598795 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.599015 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260927 04:07:23.599738 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.591922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44444)
0927 04:07:23.591970 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.591980 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.592013 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.592359 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.592360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.592522 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.592559 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.592561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.593349 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.593350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.593984 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.593987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.594561 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.594563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.594625 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.595097 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.595102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.596593 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.598581 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.598582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.598583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.598764 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.599035 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.599036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.599037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.599099 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.599437 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.599438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.599478 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.599527 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.599638 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 04:07:23.600088 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.592080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.17.184.252:44333)
0927 04:07:23.592236 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.592238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.592245 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:07:23.592263 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.592264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.592272 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.592342 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.592642 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.593223 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.594085 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.594406 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.594535 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.596706 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.596712 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.596712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.596718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.598501 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.598875 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.598878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.598879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.598957 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.599174 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.599175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.599175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.599359 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.599861 (+ 502us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.599908 (+ 47us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.599982 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":77,"threads_started":1}
I20260927 04:07:23.600139 8113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:23.600831 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.589834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.17.184.252:44333)
0927 04:07:23.590013 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.590018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.590079 (+ 61us) server_negotiation.cc:407] Connection header received
0927 04:07:23.590168 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.590174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.590208 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.590300 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.590948 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.591746 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.592694 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.593203 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.593263 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.597515 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.597516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.597516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.597523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.599223 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.599634 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.599634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.599635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.599693 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.599906 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.599907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.599907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.600039 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.600168 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.600621 (+ 453us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.600707 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260927 04:07:23.602267 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25: Bootstrap starting.
I20260927 04:07:23.603101 7963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:23.603433 7963 log.cc:824] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:23.604272 7963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25: No bootstrap required, opened a new log
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:07:23.606060 7963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } }
I20260927 04:07:23.606199 7963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70d0945c09a54be1a8523789f804be25, State: Initialized, Role: FOLLOWER
I20260927 04:07:23.606283 7963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } }
I20260927 04:07:23.607337 7963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:23.607348 8120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.607514 8120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:23.608521 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.600493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.17.184.253:44199)
0927 04:07:23.600580 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.600582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.600672 (+ 90us) server_negotiation.cc:407] Connection header received
0927 04:07:23.600707 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.600708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.600718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.600778 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.601182 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.601781 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.602633 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.602992 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.603063 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.605533 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.605535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.605536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.605544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.607174 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.607576 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.607577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.607578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.607636 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.607893 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.607894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.607894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.608047 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.608124 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.608340 (+ 216us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.608431 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 04:07:23.608945 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.600399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:44199 (local address 127.0.0.1:48080)
0927 04:07:23.600607 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.600616 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.600662 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.600781 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.600782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.601013 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.601125 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.601127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.601856 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.601857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.602541 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.602543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.603120 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.603121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.603205 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.603609 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.603615 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.605435 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.607259 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.607260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.607260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.607488 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.607747 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.607748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.607748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.607813 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.608160 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.608161 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.608209 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.608711 (+ 502us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.608825 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
I20260927 04:07:23.609515 8113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:23.609701 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.601412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.17.184.254:42633)
0927 04:07:23.601449 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.601451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.601458 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:07:23.601485 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.601486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.601494 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.601542 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.602163 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.602786 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.603774 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.604043 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.604107 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.606671 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.606674 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.606674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.606680 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.608349 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.608749 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.608751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.608751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.608806 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.609054 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.609055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.609056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.609194 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.609295 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.609524 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.609594 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 04:07:23.610776 8131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:23.610831 8131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:23.610409 7895 master_runner.cc:186] Error getting master registration for 127.17.184.253:44199: OK, instance_id { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" instance_seqno: 1790482043429267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:07:23.612740 7895 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:07:23.616981 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.609795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44460)
0927 04:07:23.609913 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.609921 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.609964 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.610081 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.610082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.610229 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.610260 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.610262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.611179 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.611180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.611900 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.611902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.612391 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.612391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.612431 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.612880 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.612885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.614018 (+ 1133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.615911 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.615913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.615913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.616123 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.616357 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.616357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.616358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.616414 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.616659 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.616659 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.616682 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.616762 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.616860 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260927 04:07:23.617482 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.609867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.17.184.252:44333)
0927 04:07:23.609895 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.609897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.609953 (+ 56us) server_negotiation.cc:407] Connection header received
0927 04:07:23.610026 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.610027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.610037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.610080 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.610320 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.611032 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.611993 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.612291 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.612331 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.614110 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.614111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.614112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.614119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.615821 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.616207 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.616208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.616208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.616289 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.616488 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.616489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.616489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.616595 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.616629 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.617329 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.617413 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260927 04:07:23.619048 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.610489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.17.184.253:44199)
0927 04:07:23.610523 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.610525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.610600 (+ 75us) server_negotiation.cc:407] Connection header received
0927 04:07:23.610703 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.610703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.610714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.610743 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.611417 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.612160 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.613193 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.613481 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.613527 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.616025 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.616027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.616027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.616035 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.617743 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.618131 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.618133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.618133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.618192 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.618405 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.618406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.618407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.618544 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.618715 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.618824 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.618963 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260927 04:07:23.619086 8113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a: Bootstrap starting.
I20260927 04:07:23.619838 8113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:23.620208 8113 log.cc:824] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:23.620548 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:23.621105 8113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a: No bootstrap required, opened a new log
I20260927 04:07:23.621832 7937 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "70d0945c09a54be1a8523789f804be25"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47742
I20260927 04:07:23.621963 7937 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 04:07:23.622005 7937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:23.622558 8113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } }
I20260927 04:07:23.622664 8113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3ae268b0ee4c30b3f9b1227cb4980a, State: Initialized, Role: FOLLOWER
I20260927 04:07:23.622715 8113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } }
I20260927 04:07:23.622937 7937 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } }
I20260927 04:07:23.623019 8113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:23.622988 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.623075 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:23.623183 7937 leader_election.cc:288] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [CANDIDATE]: Term 1 election: Requested vote from peers e5b115e3b9e74c0183e1dfe33a991b77 (127.17.184.253:44199), 9a3ae268b0ee4c30b3f9b1227cb4980a (127.17.184.252:44333)
I20260927 04:07:23.623440 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70d0945c09a54be1a8523789f804be25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a"
I20260927 04:07:23.623543 8087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:23.624869 8087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70d0945c09a54be1a8523789f804be25 in term 1.
I20260927 04:07:23.625133 7914 leader_election.cc:302] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70d0945c09a54be1a8523789f804be25, 9a3ae268b0ee4c30b3f9b1227cb4980a; no voters:
I20260927 04:07:23.625269 8120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:23.625442 8120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [term 1 LEADER]: Becoming Leader. State: Replica: 70d0945c09a54be1a8523789f804be25, State: Running, Role: LEADER
I20260927 04:07:23.625543 8120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } }
I20260927 04:07:23.626104 8120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d0945c09a54be1a8523789f804be25. Latest consensus state: current_term: 1 leader_uuid: "70d0945c09a54be1a8523789f804be25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.626188 8120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:23.626746 8154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:23.626991 8154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 04:07:23.629470 8155 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:23.629520 8155 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:23.629319 8045 master_runner.cc:186] Error getting master registration for 127.17.184.253:44199: OK, instance_id { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" instance_seqno: 1790482043429267 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:07:23.631181 8045 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:07:23.632018 8087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [term 1 FOLLOWER]: Refusing update from remote peer 70d0945c09a54be1a8523789f804be25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:07:23.632366 8120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:07:23.633651 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d0945c09a54be1a8523789f804be25. Latest consensus state: current_term: 1 leader_uuid: "70d0945c09a54be1a8523789f804be25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.633754 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:23.637255 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.624498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.17.184.252:44333)
0927 04:07:23.624536 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.624538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.624959 (+ 421us) server_negotiation.cc:407] Connection header received
0927 04:07:23.625123 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.625124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.625134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.625165 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.626342 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.627112 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.628260 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.628536 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.628585 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.633212 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.633214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.633214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.633222 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.635099 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.635526 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.635528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.635528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.635594 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.636473 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.636474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.636475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.636639 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.636742 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.637088 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.637162 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:07:23.637531 7966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.625023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.17.184.254:42633)
0927 04:07:23.625068 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.625070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.625713 (+ 643us) server_negotiation.cc:407] Connection header received
0927 04:07:23.625853 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.625854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.625864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.625894 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.627029 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.627756 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.630446 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.630772 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.630814 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.634336 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.634338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.634338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.634346 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.636368 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.636750 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.636751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.636752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.636810 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.637003 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.637005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.637005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.637149 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.637247 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.637354 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.637431 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:07:23.640437 8156 mvcc.cc:205] Tried to move back new op lower bound from 7333814450714234880 to 7333814450691293184. Current Snapshot: MvccSnapshot[applied={T|T < 7333814450714234880}]
I20260927 04:07:23.640534 8151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70d0945c09a54be1a8523789f804be25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.640623 8151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:23.640729 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70d0945c09a54be1a8523789f804be25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.640825 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:23.641570 8154 catalog_manager.cc:1347] Generated new cluster ID: a0850d1701724ae09788ae7728c10872
I20260927 04:07:23.641623 8154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 04:07:23.642334 8158 mvcc.cc:205] Tried to move back new op lower bound from 7333814450714234880 to 7333814450691293184. Current Snapshot: MvccSnapshot[applied={T|T < 7333814450714234880}]
I20260927 04:07:23.642710 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.625297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.17.184.253:44199)
0927 04:07:23.625335 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.625337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.635043 (+ 9706us) server_negotiation.cc:407] Connection header received
0927 04:07:23.635152 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.635153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.635180 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.635214 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.635911 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.636601 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.637604 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.637895 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.637971 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.639467 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.639468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.639469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.639483 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.641301 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.641702 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.641703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.641703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.641787 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.641983 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.641984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.641985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.642138 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.642235 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.642519 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.642601 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260927 04:07:23.661700 8154 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:23.662619 8154 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:23.668210 8038 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } attempt: 3
Sep 27 04:07:23 dist-test-slave-zqrn krb5kdc[7882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482043, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:07:23.677785 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.668723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.17.184.252:44333)
0927 04:07:23.668794 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.668796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.668803 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:07:23.668824 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.668825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.668833 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.668879 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.669370 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.670112 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.671139 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.671440 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.671512 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.673968 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:23.673970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:23.673970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:23.673977 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:23.675735 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.676134 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.676135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.676135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.676189 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:23.676370 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:23.676371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:23.676371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:23.676503 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:23.676561 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:23.677612 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0927 04:07:23.677690 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 04:07:23.677971 8040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:23.668503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:44333 (local address 127.0.0.1:44476)
0927 04:07:23.668589 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:23.668601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:23.668640 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:23.669033 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:23.669034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:23.669226 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:23.669275 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.669277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.670218 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.670220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.671045 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:23.671048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:23.671741 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:23.671742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:23.671802 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:23.672335 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:23.672341 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:23.673875 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:23.675844 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.675846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.675846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.676056 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.676254 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:23.676255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:23.676255 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:23.676308 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:23.676587 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:23.676588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:23.676630 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 04:07:23.676691 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 04:07:23.676786 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 04:07:23.679881 8038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77: Bootstrap starting.
I20260927 04:07:23.680624 8038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:23.680929 8038 log.cc:824] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:23.682294 8038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77: No bootstrap required, opened a new log
I20260927 04:07:23.684324 8038 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } }
I20260927 04:07:23.684451 8038 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5b115e3b9e74c0183e1dfe33a991b77, State: Initialized, Role: FOLLOWER
I20260927 04:07:23.684533 8038 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } }
I20260927 04:07:23.684926 8011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:23.684959 8154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25: Generated new TSK 0
I20260927 04:07:23.685217 8154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:23.685781 8011 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [term 1 FOLLOWER]: Refusing update from remote peer 70d0945c09a54be1a8523789f804be25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:07:23.685961 8012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70d0945c09a54be1a8523789f804be25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e5b115e3b9e74c0183e1dfe33a991b77"
I20260927 04:07:23.686064 8120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:07:23.684834 8038 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:23.686765 8012 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70d0945c09a54be1a8523789f804be25 in term 1.
W20260927 04:07:23.688690 8178 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:23.688735 8178 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:23.684834 8163 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.688895 8163 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:23.688951 8163 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70d0945c09a54be1a8523789f804be25. Latest consensus state: current_term: 1 leader_uuid: "70d0945c09a54be1a8523789f804be25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.689013 8163 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:23.689044 8163 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70d0945c09a54be1a8523789f804be25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:23.689076 8163 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:23.691552 8170 mvcc.cc:205] Tried to move back new op lower bound from 7333814450872905728 to 7333814450691293184. Current Snapshot: MvccSnapshot[applied={T|T < 7333814450714234880}]
W20260927 04:07:23.715442 8117 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.713659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44488 (local address 127.17.184.252:44333)
0927 04:07:23.713720 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.713723 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.714704 (+ 981us) server_negotiation.cc:407] Connection header received
0927 04:07:23.714817 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.714818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.714830 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.714862 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.715371 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44488: BlockingRecv error: recv got EOF from 127.0.0.1:44488 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260927 04:07:23.716274 7966 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.714074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.17.184.254:42633)
0927 04:07:23.714116 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.714118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.715680 (+ 1562us) server_negotiation.cc:407] Connection header received
0927 04:07:23.715783 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.715784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.715796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.715827 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.716208 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47748: BlockingRecv error: recv got EOF from 127.0.0.1:47748 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:07:23.717031 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.714410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.17.184.253:44199)
0927 04:07:23.714451 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.714453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.716511 (+ 2058us) server_negotiation.cc:407] Connection header received
0927 04:07:23.716611 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.716612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.716622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.716652 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.716974 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48108: BlockingRecv error: recv got EOF from 127.0.0.1:48108 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:07:23.720001 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.719190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.17.184.253:44199)
0927 04:07:23.719227 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.719229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.719460 (+ 231us) server_negotiation.cc:407] Connection header received
0927 04:07:23.719569 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.719570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.719588 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.719620 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.719956 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48120: BlockingRecv error: recv got EOF from 127.0.0.1:48120 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 04:07:23.723021 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.722229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.17.184.253:44199)
0927 04:07:23.722260 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.722262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.722506 (+ 244us) server_negotiation.cc:407] Connection header received
0927 04:07:23.722601 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.722602 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.722611 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.722639 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.722973 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48134: BlockingRecv error: recv got EOF from 127.0.0.1:48134 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260927 04:07:23.727002 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.726170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.17.184.253:44199)
0927 04:07:23.726213 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.726215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.726437 (+ 222us) server_negotiation.cc:407] Connection header received
0927 04:07:23.726536 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.726537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.726547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.726577 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.726960 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48140: BlockingRecv error: recv got EOF from 127.0.0.1:48140 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 04:07:23.733177 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.732256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48144 (local address 127.17.184.253:44199)
0927 04:07:23.732296 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.732298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.732572 (+ 274us) server_negotiation.cc:407] Connection header received
0927 04:07:23.732693 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.732694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.732706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.732738 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.733121 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48144: BlockingRecv error: recv got EOF from 127.0.0.1:48144 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 04:07:23.743528 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.742487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.17.184.253:44199)
0927 04:07:23.742532 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.742534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.742883 (+ 349us) server_negotiation.cc:407] Connection header received
0927 04:07:23.743024 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.743025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.743038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.743069 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.743458 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48150: BlockingRecv error: recv got EOF from 127.0.0.1:48150 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260927 04:07:23.761866 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.760868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48166 (local address 127.17.184.253:44199)
0927 04:07:23.760920 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.760923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.761252 (+ 329us) server_negotiation.cc:407] Connection header received
0927 04:07:23.761354 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.761355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.761367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.761397 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.761808 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48166: BlockingRecv error: recv got EOF from 127.0.0.1:48166 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260927 04:07:23.795914 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.795134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.17.184.253:44199)
0927 04:07:23.795182 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.795184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.795336 (+ 152us) server_negotiation.cc:407] Connection header received
0927 04:07:23.795411 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.795412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.795424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.795500 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.795852 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48180: BlockingRecv error: recv got EOF from 127.0.0.1:48180 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260927 04:07:23.862140 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.861305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48182 (local address 127.17.184.253:44199)
0927 04:07:23.861373 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.861375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.861485 (+ 110us) server_negotiation.cc:407] Connection header received
0927 04:07:23.861520 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.861521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.861534 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.861667 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.862081 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48182: BlockingRecv error: recv got EOF from 127.0.0.1:48182 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 04:07:23.992193 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:23.991350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.17.184.253:44199)
0927 04:07:23.991462 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:23.991464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:23.991591 (+ 127us) server_negotiation.cc:407] Connection header received
0927 04:07:23.991662 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:23.991663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:23.991676 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:23.991755 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:23.992112 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48190: BlockingRecv error: recv got EOF from 127.0.0.1:48190 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 04:07:24.062536 8120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70d0945c09a54be1a8523789f804be25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70d0945c09a54be1a8523789f804be25" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 42633 } } peers { permanent_uuid: "e5b115e3b9e74c0183e1dfe33a991b77" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 44199 } } peers { permanent_uuid: "9a3ae268b0ee4c30b3f9b1227cb4980a" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 44333 } } }
I20260927 04:07:24.062753 8120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70d0945c09a54be1a8523789f804be25 [sys.catalog]: This master's current role is: LEADER
W20260927 04:07:24.250209 8042 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:24.249422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48204 (local address 127.17.184.253:44199)
0927 04:07:24.249583 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:24.249585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:24.249597 (+ 12us) server_negotiation.cc:407] Connection header received
0927 04:07:24.249634 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:24.249635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:24.249646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:24.249708 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:24.250150 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48204: BlockingRecv error: recv got EOF from 127.0.0.1:48204 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
I20260927 04:07:24.629978 8155 catalog_manager.cc:1259] Loaded cluster ID: a0850d1701724ae09788ae7728c10872
I20260927 04:07:24.630100 8155 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a: loading cluster ID for follower catalog manager: success
I20260927 04:07:24.630991 8155 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a: acquiring CA information for follower catalog manager: success
I20260927 04:07:24.631325 8155 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9a3ae268b0ee4c30b3f9b1227cb4980a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:07:24.689234 8178 catalog_manager.cc:1259] Loaded cluster ID: a0850d1701724ae09788ae7728c10872
I20260927 04:07:24.689327 8178 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77: loading cluster ID for follower catalog manager: success
I20260927 04:07:24.690285 8178 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77: acquiring CA information for follower catalog manager: success
I20260927 04:07:24.690641 8178 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e5b115e3b9e74c0183e1dfe33a991b77: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:07:24.766484 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:24.763491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.17.184.253:44199)
0927 04:07:24.763616 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:24.763619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:24.763636 (+ 17us) server_negotiation.cc:407] Connection header received
0927 04:07:24.763757 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:24.763758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:24.763810 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:24.763926 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:24.764106 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.764658 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.765542 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.765770 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.765874 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:24.766090 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:24.766208 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:24.766306 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:24.766350 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 04:07:24.766371 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260927 04:07:24.771095 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:24.768602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.17.184.252:44333)
0927 04:07:24.768732 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:24.768735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:24.768745 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:07:24.768790 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:24.768791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:24.768842 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:24.768935 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:24.769037 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.769530 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.770274 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.770491 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.770568 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:24.770732 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:24.770843 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:24.770952 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:24.770983 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 04:07:24.771003 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260927 04:07:24.775607 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:24.772861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.17.184.252:44333)
0927 04:07:24.773016 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:24.773019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:24.773028 (+ 9us) server_negotiation.cc:407] Connection header received
0927 04:07:24.773077 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:24.773078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:24.773088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:24.773151 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:24.773265 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.773828 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.774771 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.775081 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.775146 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:24.775343 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:24.775375 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:24.775443 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:24.775504 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 04:07:24.775524 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 04:07:24.775974 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:24.772984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60328 (local address 127.17.184.253:44199)
0927 04:07:24.773026 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:24.773028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:24.773154 (+ 126us) server_negotiation.cc:407] Connection header received
0927 04:07:24.773234 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:24.773235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:24.773246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:24.773293 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:24.773456 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.774187 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.775176 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.775467 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.775526 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:24.775711 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:24.775744 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:24.775773 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:24.775886 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 04:07:24.775912 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:07:24.776360 8261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:24.772919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43944 (local address 127.17.184.254:42633)
0927 04:07:24.773085 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:24.773087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:24.773196 (+ 109us) server_negotiation.cc:407] Connection header received
0927 04:07:24.773241 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:24.773242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:24.773289 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:24.773369 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:24.773500 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.774282 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.775235 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:24.775540 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:24.775595 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:24.775957 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:24.776064 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:24.776124 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:24.776223 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 04:07:24.776254 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":54,"threads_started":1}
I20260927 04:07:24.778100 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7895
I20260927 04:07:24.783417 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 7970
I20260927 04:07:24.787976 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 8045
2026-09-27T04:07:24Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2870 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 04:07:24 dist-test-slave-zqrn krb5kdc[8265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 04:07:24 dist-test-slave-zqrn krb5kdc[8265](info): set up 2 sockets
Sep 27 04:07:24 dist-test-slave-zqrn krb5kdc[8265](info): commencing operation
krb5kdc: starting...
W20260927 04:07:25.831228 18147 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/test-user.keytab.
Sep 27 04:07:25 dist-test-slave-zqrn krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T04:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T04:07:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.184.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
Entry for principal kudu/127.17.184.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab.
I20260927 04:07:25.953680 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:43669
--webserver_interface=127.17.184.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:43669,127.17.184.253:46839,127.17.184.252:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 04:07:26.038643 8278 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:07:26.038887 8278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:26.038923 8278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:26.038949 8278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:26.040314 8278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:26.040369 8278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:26.040385 8278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:26.040397 8278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:26.040411 8278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:26.042188 8278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:43669,127.17.184.253:46839,127.17.184.252:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.254.keytab
--principal=kudu/127.17.184.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.254:43669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.184.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:26.042490 8278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:26.042768 8278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:26.048221 8286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:26.048206 8284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:26.048195 8283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:26.048476 8278 server_base.cc:1034] running on GCE node
I20260927 04:07:26.048804 8278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:26.049188 8278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:26.050379 8278 hybrid_clock.cc:648] HybridClock initialized: now 1790482046050349 us; error 47 us; skew 500 ppm
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:26.059028 8278 init.cc:377] Logged in from keytab as kudu/127.17.184.254@KRBTEST.COM (short username kudu)
I20260927 04:07:26.059585 8278 webserver.cc:460] Webserver started at http://127.17.184.254:42623/ using document root <none> and password file <none>
I20260927 04:07:26.059837 8278 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:26.059891 8278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:26.060041 8278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:26.061219 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/data/instance:
uuid: "cd71d5f4301344d9b9c7fff971891c88"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "847365e6bfcbabbf611df7fc55b32cfd"
server_key_iv: "cb50eb51c26ab7e2affcc3c8b8c5e52a"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.061592 8278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/wal/instance:
uuid: "cd71d5f4301344d9b9c7fff971891c88"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "847365e6bfcbabbf611df7fc55b32cfd"
server_key_iv: "cb50eb51c26ab7e2affcc3c8b8c5e52a"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.063107 8278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260927 04:07:26.064051 8293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:26.064455 8278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:26.064561 8278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/wal
uuid: "cd71d5f4301344d9b9c7fff971891c88"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "847365e6bfcbabbf611df7fc55b32cfd"
server_key_iv: "cb50eb51c26ab7e2affcc3c8b8c5e52a"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.064658 8278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:26.084426 8278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:26.087088 8278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:26.087260 8278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:26.091830 8278 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.254:43669
I20260927 04:07:26.091864 8345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.254:43669 every 8 connection(s)
I20260927 04:07:26.092267 8278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/data/info.pb
I20260927 04:07:26.093158 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:26.094718 8346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:26.099176 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 8278
I20260927 04:07:26.099274 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-0/wal/instance
I20260927 04:07:26.099480 18147 external_mini_cluster.cc:1428] Setting key ae594fcc95e181954b37ddd67f9906d7
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:07:26.106026 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.095422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:43669 (local address 127.0.0.1:51746)
0927 04:07:26.095589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.095632 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.096103 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.096367 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.096369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.096620 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.096780 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.096783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.097756 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.097757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.098837 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.098840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.099683 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.099684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.099751 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.100336 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.100346 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.102241 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.104343 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.104349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.104351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.104538 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.104890 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.104891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.104891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.104989 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.105481 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.105488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.105580 (+ 92us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.105676 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.105828 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260927 04:07:26.106105 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.095482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51746 (local address 127.17.184.254:43669)
0927 04:07:26.095633 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.095644 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.095726 (+ 82us) server_negotiation.cc:407] Connection header received
0927 04:07:26.096235 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.096237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.096255 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.096342 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.096872 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.097674 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.098932 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.099471 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.099586 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.102335 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.102342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.102342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.102349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.104264 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.104681 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.104684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.104685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.104791 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.105066 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.105067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.105068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.105382 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.105482 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.105743 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.105917 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":120,"thread_start_us":149,"threads_started":1}
I20260927 04:07:26.106964 8346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:26.107403 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.107242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43094)
0927 04:07:26.107303 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.107371 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 04:07:26.107566 8297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
I20260927 04:07:26.108040 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.107940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43098)
0927 04:07:26.107993 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.108022 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 04:07:26.108174 8346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:46839: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.17.184.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab.
Entry for principal kudu/127.17.184.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab.
I20260927 04:07:26.126571 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.253:46839
--webserver_interface=127.17.184.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab
--principal=kudu/127.17.184.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:43669,127.17.184.253:46839,127.17.184.252:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:07:26.157478 8346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } attempt: 1
I20260927 04:07:26.158119 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.157953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43106)
0927 04:07:26.158040 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.158085 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 04:07:26.158540 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.158450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43114)
0927 04:07:26.158494 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.158520 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:07:26.158711 8346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:46839: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
W20260927 04:07:26.208248 8353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:07:26.208468 8353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:26.208485 8353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:26.208549 8353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:26.209755 8353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:26.209795 8353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:26.209810 8353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:26.209822 8353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:26.209874 8353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:26.211997 8353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:43669,127.17.184.253:46839,127.17.184.252:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.253.keytab
--principal=kudu/127.17.184.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.253:46839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.184.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:26.212284 8353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:26.212549 8353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:26.217512 8361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:26.217463 8358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:26.217475 8359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:26.217762 8353 server_base.cc:1034] running on GCE node
I20260927 04:07:26.218034 8353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:26.218343 8353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:26.219520 8353 hybrid_clock.cc:648] HybridClock initialized: now 1790482046219482 us; error 53 us; skew 500 ppm
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:26.222092 8353 init.cc:377] Logged in from keytab as kudu/127.17.184.253@KRBTEST.COM (short username kudu)
I20260927 04:07:26.222579 8353 webserver.cc:460] Webserver started at http://127.17.184.253:41557/ using document root <none> and password file <none>
I20260927 04:07:26.222836 8353 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:26.222882 8353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:26.223116 8353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:26.224217 8353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/data/instance:
uuid: "b089665079014679aa6a64c4eb3cc03c"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "2c8d93b5c6b65d4eec651bd92528692c"
server_key_iv: "685f9735a28b5d95fd9e75ec6cb54a87"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.224601 8353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/wal/instance:
uuid: "b089665079014679aa6a64c4eb3cc03c"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "2c8d93b5c6b65d4eec651bd92528692c"
server_key_iv: "685f9735a28b5d95fd9e75ec6cb54a87"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.225975 8353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 04:07:26.226780 8368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:26.226879 8346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } attempt: 2
I20260927 04:07:26.227023 8353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 04:07:26.227137 8353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/wal
uuid: "b089665079014679aa6a64c4eb3cc03c"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "2c8d93b5c6b65d4eec651bd92528692c"
server_key_iv: "685f9735a28b5d95fd9e75ec6cb54a87"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.227227 8353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:26.227337 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.227212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43122)
0927 04:07:26.227268 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.227301 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 04:07:26.227669 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.227574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43126)
0927 04:07:26.227621 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.227645 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:07:26.227766 8346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.253:46839: Network error: Client connection negotiation failed: client connection to 127.17.184.253:46839: connect: Connection refused (error 111)
I20260927 04:07:26.239799 8353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:26.241930 8353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:26.242108 8353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:26.246170 8353 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.253:46839
I20260927 04:07:26.246241 8420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.253:46839 every 8 connection(s)
I20260927 04:07:26.246585 8353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/data/info.pb
I20260927 04:07:26.247493 8421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:26.249054 8421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:26.251434 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 8353
I20260927 04:07:26.251550 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-1/wal/instance
I20260927 04:07:26.251744 18147 external_mini_cluster.cc:1428] Setting key 06a7b99fec9c7764c64f31f30f024306
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:07:26.258647 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.249813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51750 (local address 127.17.184.254:43669)
0927 04:07:26.249910 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.249912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.250083 (+ 171us) server_negotiation.cc:407] Connection header received
0927 04:07:26.250622 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.250624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.250638 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.250696 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.251385 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.252026 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.253024 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.253243 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.253326 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.255851 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.255853 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.255854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.255864 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.257265 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.257686 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.257687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.257688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.257748 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.258000 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.258001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.258002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.258117 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.258189 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.258470 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.258553 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 04:07:26.258836 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.249732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:43669 (local address 127.0.0.1:51750)
0927 04:07:26.249952 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.249992 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.250513 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.250773 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.250776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.251089 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.251289 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.251293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.252117 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.252118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.252915 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.252917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.253337 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.253339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.253459 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.254013 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.254024 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.255738 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.257366 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.257373 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.257375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.257600 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.257818 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.257819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.257819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.257931 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.258199 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.258207 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.258341 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.258450 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.258605 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.17.184.252@KRBTEST.COM; defaulting to no policy
I20260927 04:07:26.259392 8421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.184.252@KRBTEST.COM" created.
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:07:26.267640 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.259696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43136)
0927 04:07:26.259752 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.259764 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.259807 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.260118 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.260119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.260266 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.260308 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.260309 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.261281 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.261282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.261776 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.261778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.262279 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.262280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.262343 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.262819 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.262825 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.264406 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.266523 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.266524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.266525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.266727 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.266984 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.266984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.266985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.267051 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.267390 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.267391 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.267426 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.267475 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.267572 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 04:07:26.267813 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.259816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.17.184.253:46839)
0927 04:07:26.259983 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.259987 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.260002 (+ 15us) server_negotiation.cc:407] Connection header received
0927 04:07:26.260018 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.260019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.260030 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.260118 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.260384 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.261176 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.261872 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.262191 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.262280 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.264494 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.264501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.264501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.264507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.266425 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.266815 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.266818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.266819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.266883 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.267118 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.267119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.267120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.267321 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.267417 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.267516 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.267644 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260927 04:07:26.268464 8421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:26.268863 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.268744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44292)
0927 04:07:26.268792 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.268835 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 04:07:26.269032 8370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
I20260927 04:07:26.269512 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.269410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44300)
0927 04:07:26.269459 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.269482 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 04:07:26.269680 8421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:40863: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.184.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab.
Entry for principal kudu/127.17.184.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab.
I20260927 04:07:26.281457 18147 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskJqOnmp/build/release/bin/kudu
/tmp/dist-test-taskJqOnmp/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.252:40863
--webserver_interface=127.17.184.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab
--principal=kudu/127.17.184.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.184.212:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.184.254:43669,127.17.184.253:46839,127.17.184.252:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 04:07:26.318938 8346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } attempt: 3
I20260927 04:07:26.318987 8421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } attempt: 1
I20260927 04:07:26.319459 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.319320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44306)
0927 04:07:26.319380 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.319418 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:07:26.319828 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.319741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44314)
0927 04:07:26.319777 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.319800 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260927 04:07:26.319973 8421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:40863: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:07:26.326349 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.319366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43146 (local address 127.17.184.253:46839)
0927 04:07:26.319415 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.319417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.319427 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:07:26.319497 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.319498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.319576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.319654 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.320072 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.320578 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.321294 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.321518 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.321599 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.323744 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.323746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.323747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.323755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.325230 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.325666 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.325666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.325667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.325716 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.325910 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.325911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.325911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.326013 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.326069 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.326223 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.326285 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 04:07:26.326448 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.319320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43146)
0927 04:07:26.319380 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.319392 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.319437 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.319684 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.319686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.319891 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.319971 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.319974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.320670 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.320671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.321206 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.321208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.321597 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.321598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.321683 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.322150 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.322155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.323649 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.325351 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.325352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.325352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.325592 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.325782 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.325783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.325783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.325852 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.326077 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.326078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.326129 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.326199 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.326304 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 04:07:26.326877 8346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:26.327211 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.327125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44318)
0927 04:07:26.327159 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.327188 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260927 04:07:26.327610 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.327484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44320)
0927 04:07:26.327523 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.327593 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 04:07:26.327718 8346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:40863: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
I20260927 04:07:26.358860 8346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } attempt: 1
I20260927 04:07:26.359453 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.359291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44324)
0927 04:07:26.359377 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.359421 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 04:07:26.359946 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.359849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44326)
0927 04:07:26.359895 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.359923 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:07:26.360075 8346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:40863: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
W20260927 04:07:26.364859 8428 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 04:07:26.365056 8428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 04:07:26.365073 8428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 04:07:26.365131 8428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 04:07:26.366272 8428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 04:07:26.366312 8428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 04:07:26.366325 8428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 04:07:26.366338 8428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 04:07:26.366349 8428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 04:07:26.367843 8428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.184.212:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.184.254:43669,127.17.184.253:46839,127.17.184.252:40863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/krb5kdc/kudu_127.17.184.252.keytab
--principal=kudu/127.17.184.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.184.252:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.184.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 27 Sep 2026 02:47:41 UTC on ccycloud.quasar-xcazef.root.comops.site
build id 4675619
I20260927 04:07:26.368075 8428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 04:07:26.368284 8428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 04:07:26.373199 8434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:26.373310 8428 server_base.cc:1034] running on GCE node
W20260927 04:07:26.373199 8433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 04:07:26.373251 8436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 04:07:26.373762 8428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 04:07:26.374059 8428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 04:07:26.375223 8428 hybrid_clock.cc:648] HybridClock initialized: now 1790482046375205 us; error 39 us; skew 500 ppm
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 04:07:26.377907 8428 init.cc:377] Logged in from keytab as kudu/127.17.184.252@KRBTEST.COM (short username kudu)
I20260927 04:07:26.378368 8428 webserver.cc:460] Webserver started at http://127.17.184.252:43545/ using document root <none> and password file <none>
I20260927 04:07:26.378643 8428 fs_manager.cc:359] Metadata directory not provided
I20260927 04:07:26.378690 8428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 04:07:26.378861 8428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 04:07:26.379971 8428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/data/instance:
uuid: "2ec860bd707a49f9a97d52d8fe670288"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "66188b2878076d4b0054df2ded9d29e5"
server_key_iv: "65f171dfbcd8f86ae94b33b5b94e2bad"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.380349 8428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/wal/instance:
uuid: "2ec860bd707a49f9a97d52d8fe670288"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "66188b2878076d4b0054df2ded9d29e5"
server_key_iv: "65f171dfbcd8f86ae94b33b5b94e2bad"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.381737 8428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260927 04:07:26.382519 8443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 04:07:26.382730 8428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 04:07:26.382839 8428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/data,/tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/wal
uuid: "2ec860bd707a49f9a97d52d8fe670288"
format_stamp: "Formatted at 2026-09-27 04:07:26 on dist-test-slave-zqrn"
server_key: "66188b2878076d4b0054df2ded9d29e5"
server_key_iv: "65f171dfbcd8f86ae94b33b5b94e2bad"
server_key_version: "encryptionkey@0"
I20260927 04:07:26.382943 8428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 04:07:26.388136 8421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } attempt: 2
I20260927 04:07:26.388582 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.388454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44328)
0927 04:07:26.388513 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.388551 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:07:26.388963 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.388876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44342)
0927 04:07:26.388919 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.388945 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 04:07:26.389068 8421 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.184.252:40863: Network error: Client connection negotiation failed: client connection to 127.17.184.252:40863: connect: Connection refused (error 111)
I20260927 04:07:26.394611 8428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 04:07:26.396937 8428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 04:07:26.397105 8428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 04:07:26.401086 8428 rpc_server.cc:307] RPC server started. Bound to: 127.17.184.252:40863
I20260927 04:07:26.401134 8495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.184.252:40863 every 8 connection(s)
I20260927 04:07:26.401521 8428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/data/info.pb
I20260927 04:07:26.402375 8496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 04:07:26.403995 8496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:26.406718 18147 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskJqOnmp/build/release/bin/kudu as pid 8428
I20260927 04:07:26.406934 18147 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskJqOnmp/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790481773036938-18147-0/minicluster-data/master-2/wal/instance
I20260927 04:07:26.407179 18147 external_mini_cluster.cc:1428] Setting key 4c32a102522d47612a7ef507c7b703cf
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:07:26.414868 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.404781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51760 (local address 127.17.184.254:43669)
0927 04:07:26.404854 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.404856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.405027 (+ 171us) server_negotiation.cc:407] Connection header received
0927 04:07:26.405536 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.405538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.405549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.405619 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.406258 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.407166 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.408510 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.408793 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.408894 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.411410 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.411412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.411413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.411421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.413009 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.413537 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.413539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.413539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.413603 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.414004 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.414006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.414006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.414154 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.414242 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.414663 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.414754 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 04:07:26.415122 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.404656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.254:43669 (local address 127.0.0.1:51760)
0927 04:07:26.404887 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.404937 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.405428 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.405683 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.405687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.405943 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.406167 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.406170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.407303 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.407304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.408384 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.408386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.408903 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.408905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.409034 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.409572 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.409581 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.411308 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.413148 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.413155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.413157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.413429 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.413692 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.413692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.413693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.413911 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.414257 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.414264 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.414498 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.414627 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.414796 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260927 04:07:26.415786 8496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:26.416766 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.407691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44356 (local address 127.17.184.252:40863)
0927 04:07:26.407891 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.407897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.407947 (+ 50us) server_negotiation.cc:407] Connection header received
0927 04:07:26.407971 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.407974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.407994 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.408123 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.408878 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.409581 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.410448 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.410735 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.410810 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.413669 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.413677 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.413677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.413685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.415525 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.415882 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.415890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.415890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.415974 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.416178 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.416179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.416180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.416368 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.416542 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.416600 (+ 58us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.416675 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":135,"threads_started":1}
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.254@KRBTEST.COM
I20260927 04:07:26.422793 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.416081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.253:46839 (local address 127.0.0.1:43162)
0927 04:07:26.416125 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.416134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.416168 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.416312 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.416313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.416468 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.416503 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.416505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.417322 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.417323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.418033 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.418036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.418417 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.418418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.418479 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.418851 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.418856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.420066 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.421719 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.421720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.421720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.421941 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.422156 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.422157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.422157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.422218 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.422480 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.422481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.422526 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.422590 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.422697 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 04:07:26.422828 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.416162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.17.184.253:46839)
0927 04:07:26.416211 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.416212 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.416225 (+ 13us) server_negotiation.cc:407] Connection header received
0927 04:07:26.416246 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.416247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.416255 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.416297 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.416585 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.417229 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.418114 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.418349 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.418400 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.420153 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.420155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.420155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.420169 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.421605 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.422023 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.422024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.422025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.422079 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.422287 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.422288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.422288 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.422399 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.422486 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.422669 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.422740 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 04:07:26.423341 8496 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
I20260927 04:07:26.425088 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.417519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.17.184.254:43669)
0927 04:07:26.417555 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.417557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.417580 (+ 23us) server_negotiation.cc:407] Connection header received
0927 04:07:26.417638 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.417639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.417647 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.417702 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.418148 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.418721 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.419474 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.419681 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.419764 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.422213 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.422214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.422214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.422221 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.423961 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.424318 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.424319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.424319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.424380 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.424558 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.424559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.424559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.424690 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.424861 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.424898 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.424977 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.252@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.184.253@KRBTEST.COM
I20260927 04:07:26.430478 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.423725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44358)
0927 04:07:26.423772 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.423781 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.423818 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.423933 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.423934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.424082 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.424118 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.424119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.424950 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.424951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.425630 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.425632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.426091 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.426092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.426174 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.426550 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.426554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.427718 (+ 1164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.429431 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.429431 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.429432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.429630 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.429820 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.429821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.429821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.429884 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.430149 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.430151 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.430203 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.430283 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.430385 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 04:07:26.430603 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.423787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.17.184.252:40863)
0927 04:07:26.423823 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.423825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.423832 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:07:26.423870 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.423871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.423881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.423922 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.424184 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.424874 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.425740 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.426020 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.426069 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.427774 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.427775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.427776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.427782 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.429324 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.429699 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.429700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.429700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.429749 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.429944 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.429944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.429945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.430049 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.430137 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.430309 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.430472 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":22}
I20260927 04:07:26.432394 8496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288: Bootstrap starting.
I20260927 04:07:26.433154 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.425737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.17.184.253:46839)
0927 04:07:26.425771 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.425773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.425784 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:07:26.425818 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.425818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.425827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.425877 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.426398 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.426977 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.427690 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.427991 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.428054 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.430365 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.430367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.430367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.430373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.432062 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.432396 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.432397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.432397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.432454 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.432691 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.432692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.432692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.432797 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.432928 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.433038 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.433093 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:07:26.433480 8496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:26.433772 8496 log.cc:824] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:26.434597 8496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288: No bootstrap required, opened a new log
I20260927 04:07:26.435171 18147 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 04:07:26.435230 8346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } attempt: 2
I20260927 04:07:26.436748 8496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } }
I20260927 04:07:26.436903 8496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec860bd707a49f9a97d52d8fe670288, State: Initialized, Role: FOLLOWER
I20260927 04:07:26.436281 8320 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cd71d5f4301344d9b9c7fff971891c88"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51776
I20260927 04:07:26.436978 8496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } }
I20260927 04:07:26.437289 8496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:26.437249 8502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.437345 8502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 04:07:26.439132 8513 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:26.439231 8513 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.254@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:07:26.440097 8428 master_runner.cc:186] Error getting master registration for 127.17.184.254:43669: OK, instance_id { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" instance_seqno: 1790482046090611 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:07:26.442850 8428 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:07:26.443816 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.436014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.17.184.252:40863)
0927 04:07:26.436083 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.436084 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.436111 (+ 27us) server_negotiation.cc:407] Connection header received
0927 04:07:26.436130 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.436131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.436140 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.436183 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.436522 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.437208 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.438017 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.438314 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.438374 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.440574 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.440576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.440577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.440585 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.442450 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.442878 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.442879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.442880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.442965 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.443206 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.443207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.443207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.443372 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.443439 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.443636 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.443719 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 04:07:26.443830 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.435951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44362)
0927 04:07:26.436010 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.436022 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.436071 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.436222 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.436223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.436391 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.436438 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.436440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.437294 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.437295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.437919 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.437922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.438373 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.438374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.438431 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.438883 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.438939 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.440483 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.442550 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.442551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.442552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.442788 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.443029 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.443030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.443031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.443127 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.443455 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.443456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.443514 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.443603 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.443719 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 04:07:26.445593 8346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88: Bootstrap starting.
I20260927 04:07:26.446429 8346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:26.446780 8346 log.cc:824] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:26.447749 8346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88: No bootstrap required, opened a new log
I20260927 04:07:26.449318 8346 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } }
I20260927 04:07:26.449478 8346 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd71d5f4301344d9b9c7fff971891c88, State: Initialized, Role: FOLLOWER
I20260927 04:07:26.449584 8346 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } }
I20260927 04:07:26.449956 8346 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:26.449910 8516 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.450003 8320 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 04:07:26.450029 8516 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:26.450060 8320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:26.450784 8320 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } }
I20260927 04:07:26.451593 8470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd71d5f4301344d9b9c7fff971891c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2ec860bd707a49f9a97d52d8fe670288"
I20260927 04:07:26.451735 8470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [term 0 FOLLOWER]: Advancing to term 1
W20260927 04:07:26.452389 8527 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:26.452458 8527 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:26.452737 8470 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd71d5f4301344d9b9c7fff971891c88 in term 1.
I20260927 04:07:26.453373 8295 leader_election.cc:302] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ec860bd707a49f9a97d52d8fe670288, cd71d5f4301344d9b9c7fff971891c88; no voters:
I20260927 04:07:26.453503 8516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 04:07:26.453826 8516 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [term 1 LEADER]: Becoming Leader. State: Replica: cd71d5f4301344d9b9c7fff971891c88, State: Running, Role: LEADER
I20260927 04:07:26.453960 8516 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } }
I20260927 04:07:26.454434 8516 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd71d5f4301344d9b9c7fff971891c88. Latest consensus state: current_term: 1 leader_uuid: "cd71d5f4301344d9b9c7fff971891c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.454526 8516 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:26.453833 8278 master_runner.cc:186] Error getting master registration for 127.17.184.253:46839: OK, instance_id { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" instance_seqno: 1790482046245079 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 04:07:26.456271 8278 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 04:07:26.453845 8320 leader_election.cc:288] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [CANDIDATE]: Term 1 election: Requested vote from peers b089665079014679aa6a64c4eb3cc03c (127.17.184.253:46839), 2ec860bd707a49f9a97d52d8fe670288 (127.17.184.252:40863)
I20260927 04:07:26.459707 8531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 04:07:26.459946 8531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 04:07:26.464249 8470 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [term 1 FOLLOWER]: Refusing update from remote peer cd71d5f4301344d9b9c7fff971891c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:07:26.464689 8516 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 04:07:26.466176 8502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd71d5f4301344d9b9c7fff971891c88. Latest consensus state: current_term: 1 leader_uuid: "cd71d5f4301344d9b9c7fff971891c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.466599 8502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:26.469651 8349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.457457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.17.184.254:43669)
0927 04:07:26.457498 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.457501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.458178 (+ 677us) server_negotiation.cc:407] Connection header received
0927 04:07:26.458356 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.458369 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.458380 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.458414 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.460008 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.460704 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.462222 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.462520 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.462559 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.464883 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.464885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.464885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.464894 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.467016 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.468449 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.468450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.468451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.468512 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.468865 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.468866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.468867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.469026 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.469128 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.469465 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.469550 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260927 04:07:26.472177 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.457532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.17.184.253:46839)
0927 04:07:26.457581 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.457583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.458139 (+ 556us) server_negotiation.cc:407] Connection header received
0927 04:07:26.458250 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.458251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.458262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.458293 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.459242 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.459991 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.461123 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.461434 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.461481 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.465360 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.465363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.465363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.465372 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.467808 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.471088 (+ 3280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.471090 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.471091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.471157 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.471443 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.471444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.471444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.471594 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.471693 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.471984 (+ 291us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.472076 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:07:26.472448 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd71d5f4301344d9b9c7fff971891c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.472568 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: This master's current role is: LEADER
I20260927 04:07:26.473125 8502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd71d5f4301344d9b9c7fff971891c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.473261 8502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:26.473302 8539 mvcc.cc:205] Tried to move back new op lower bound from 7333814462313357312 to 7333814462276182016. Current Snapshot: MvccSnapshot[applied={T|T < 7333814462313357312}]
I20260927 04:07:26.473405 8540 mvcc.cc:205] Tried to move back new op lower bound from 7333814462313357312 to 7333814462276182016. Current Snapshot: MvccSnapshot[applied={T|T < 7333814462313357312}]
I20260927 04:07:26.475755 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.457851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.17.184.252:40863)
0927 04:07:26.457889 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.457891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.466267 (+ 8376us) server_negotiation.cc:407] Connection header received
0927 04:07:26.466363 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.466364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.466377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.466411 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.467388 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.468170 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.469234 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.469553 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.469597 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.472116 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.472129 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.472129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.472137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.474441 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.474868 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.474870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.474870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.474977 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.475194 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.475195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.475195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.475323 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.475412 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.475568 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.475651 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:07:26.480182 8421 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } attempt: 3
I20260927 04:07:26.483520 8531 catalog_manager.cc:1347] Generated new cluster ID: 58764aa4a7624cbab0fdb62d6c0d6261
I20260927 04:07:26.483592 8531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 27 04:07:26 dist-test-slave-zqrn krb5kdc[8265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790482046, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.184.253@KRBTEST.COM for kudu/127.17.184.252@KRBTEST.COM
I20260927 04:07:26.496470 8531 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 04:07:26.497236 8531 catalog_manager.cc:1504] Loading token signing keys...
I20260927 04:07:26.497332 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.480474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.184.252:40863 (local address 127.0.0.1:44382)
0927 04:07:26.480560 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0927 04:07:26.480570 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 04:07:26.480608 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 04:07:26.481102 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 04:07:26.481103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 04:07:26.481278 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 04:07:26.481328 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.481330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.482276 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.482279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.483192 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 04:07:26.483195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.491137 (+ 7942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.491139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 04:07:26.491218 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.491881 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 04:07:26.491887 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 04:07:26.493460 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 04:07:26.495726 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.495728 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.495729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.495977 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.496307 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 04:07:26.496308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 04:07:26.496308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0927 04:07:26.496382 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 04:07:26.496994 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 04:07:26.496996 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 04:07:26.497057 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 04:07:26.497118 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 04:07:26.497227 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 04:07:26.498174 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:26.480697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44382 (local address 127.17.184.252:40863)
0927 04:07:26.480970 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.480972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.480979 (+ 7us) server_negotiation.cc:407] Connection header received
0927 04:07:26.480999 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.481000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.481009 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.481056 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.481410 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.482172 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.483280 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:26.483571 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:26.483740 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:26.493557 (+ 9817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 04:07:26.493559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 04:07:26.493560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 04:07:26.493569 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 04:07:26.495470 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.496061 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.496063 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.496063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.496125 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 04:07:26.496464 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 04:07:26.496465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 04:07:26.496466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 04:07:26.496853 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 04:07:26.497765 (+ 912us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:26.497843 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 04:07:26.497986 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260927 04:07:26.500042 8421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c: Bootstrap starting.
I20260927 04:07:26.500810 8421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c: Neither blocks nor log segments found. Creating new log.
I20260927 04:07:26.501102 8421 log.cc:824] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c: Log is configured to *not* fsync() on all Append() calls
I20260927 04:07:26.505367 8531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88: Generated new TSK 0
I20260927 04:07:26.505592 8531 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 04:07:26.513013 8421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c: No bootstrap required, opened a new log
I20260927 04:07:26.515743 8421 raft_consensus.cc:348] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } }
I20260927 04:07:26.515916 8421 raft_consensus.cc:729] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b089665079014679aa6a64c4eb3cc03c, State: Initialized, Role: FOLLOWER
I20260927 04:07:26.516008 8421 consensus_queue.cc:260] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } }
I20260927 04:07:26.516366 8421 sys_catalog.cc:564] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 04:07:26.516444 8395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cd71d5f4301344d9b9c7fff971891c88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b089665079014679aa6a64c4eb3cc03c"
I20260927 04:07:26.516570 8395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [term 0 FOLLOWER]: Advancing to term 1
I20260927 04:07:26.517477 8395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd71d5f4301344d9b9c7fff971891c88 in term 1.
I20260927 04:07:26.517746 8394 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [term 1 FOLLOWER]: Refusing update from remote peer cd71d5f4301344d9b9c7fff971891c88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 04:07:26.518477 8529 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 04:07:26.518736 8558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 04:07:26.518793 8558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 04:07:26.519048 8546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.519142 8546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:26.520402 8546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd71d5f4301344d9b9c7fff971891c88. Latest consensus state: current_term: 1 leader_uuid: "cd71d5f4301344d9b9c7fff971891c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.520486 8546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:26.520524 8546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cd71d5f4301344d9b9c7fff971891c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.520561 8546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 04:07:26.527530 8561 mvcc.cc:205] Tried to move back new op lower bound from 7333814462480949248 to 7333814462276182016. Current Snapshot: MvccSnapshot[applied={T|T < 7333814462313357312}]
W20260927 04:07:26.536062 8349 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.527386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.17.184.254:43669)
0927 04:07:26.527423 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.527425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.535251 (+ 7826us) server_negotiation.cc:407] Connection header received
0927 04:07:26.535411 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.535412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.535424 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.535453 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.535987 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51792: BlockingRecv error: recv got EOF from 127.0.0.1:51792 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 04:07:26.536902 8500 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.527597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.17.184.252:40863)
0927 04:07:26.527679 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.527681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.535147 (+ 7466us) server_negotiation.cc:407] Connection header received
0927 04:07:26.536431 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.536432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.536444 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.536474 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.536838 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44396: BlockingRecv error: recv got EOF from 127.0.0.1:44396 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 04:07:26.537808 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.527163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43190 (local address 127.17.184.253:46839)
0927 04:07:26.527209 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.527212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.537221 (+ 10009us) server_negotiation.cc:407] Connection header received
0927 04:07:26.537405 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.537406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.537416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.537445 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.537740 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43190: BlockingRecv error: recv got EOF from 127.0.0.1:43190 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260927 04:07:26.540359 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.539639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43198 (local address 127.17.184.253:46839)
0927 04:07:26.539684 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.539686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.539874 (+ 188us) server_negotiation.cc:407] Connection header received
0927 04:07:26.539974 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.539975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.539992 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.540021 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.540313 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43198: BlockingRecv error: recv got EOF from 127.0.0.1:43198 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260927 04:07:26.543519 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.542676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.17.184.253:46839)
0927 04:07:26.542711 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.542714 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.543052 (+ 338us) server_negotiation.cc:407] Connection header received
0927 04:07:26.543146 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.543147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.543156 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.543184 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.543473 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43200: BlockingRecv error: recv got EOF from 127.0.0.1:43200 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260927 04:07:26.547459 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.546639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43212 (local address 127.17.184.253:46839)
0927 04:07:26.546691 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.546693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.546951 (+ 258us) server_negotiation.cc:407] Connection header received
0927 04:07:26.547047 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.547048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.547058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.547086 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.547415 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43212: BlockingRecv error: recv got EOF from 127.0.0.1:43212 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 04:07:26.553555 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.552701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.17.184.253:46839)
0927 04:07:26.552740 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.552742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.553009 (+ 267us) server_negotiation.cc:407] Connection header received
0927 04:07:26.553112 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.553113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.553125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.553155 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.553498 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43226: BlockingRecv error: recv got EOF from 127.0.0.1:43226 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260927 04:07:26.563941 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.563021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43240 (local address 127.17.184.253:46839)
0927 04:07:26.563068 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.563070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.563344 (+ 274us) server_negotiation.cc:407] Connection header received
0927 04:07:26.563447 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.563448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.563459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.563490 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.563880 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43240: BlockingRecv error: recv got EOF from 127.0.0.1:43240 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 04:07:26.582307 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.581366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43256 (local address 127.17.184.253:46839)
0927 04:07:26.581413 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.581415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.581715 (+ 300us) server_negotiation.cc:407] Connection header received
0927 04:07:26.581821 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.581823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.581835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.581866 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.582245 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43256: BlockingRecv error: recv got EOF from 127.0.0.1:43256 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 04:07:26.616962 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.615952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.17.184.253:46839)
0927 04:07:26.615999 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.616002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.616312 (+ 310us) server_negotiation.cc:407] Connection header received
0927 04:07:26.616421 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.616423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.616434 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.616466 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.616898 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43264: BlockingRecv error: recv got EOF from 127.0.0.1:43264 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260927 04:07:26.683418 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.682545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43270 (local address 127.17.184.253:46839)
0927 04:07:26.682741 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.682743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.682754 (+ 11us) server_negotiation.cc:407] Connection header received
0927 04:07:26.682783 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.682784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.682800 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.682874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.683331 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43270: BlockingRecv error: recv got EOF from 127.0.0.1:43270 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
W20260927 04:07:26.813462 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:26.812740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.17.184.253:46839)
0927 04:07:26.812799 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:26.812808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:26.812869 (+ 61us) server_negotiation.cc:407] Connection header received
0927 04:07:26.812928 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:26.812929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:26.812940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:26.812982 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:26.813375 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43278: BlockingRecv error: recv got EOF from 127.0.0.1:43278 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 04:07:26.947777 8529 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cd71d5f4301344d9b9c7fff971891c88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd71d5f4301344d9b9c7fff971891c88" member_type: VOTER last_known_addr { host: "127.17.184.254" port: 43669 } } peers { permanent_uuid: "b089665079014679aa6a64c4eb3cc03c" member_type: VOTER last_known_addr { host: "127.17.184.253" port: 46839 } } peers { permanent_uuid: "2ec860bd707a49f9a97d52d8fe670288" member_type: VOTER last_known_addr { host: "127.17.184.252" port: 40863 } } }
I20260927 04:07:26.947965 8529 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd71d5f4301344d9b9c7fff971891c88 [sys.catalog]: This master's current role is: LEADER
W20260927 04:07:27.071604 8425 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 04:07:27.070935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.17.184.253:46839)
0927 04:07:27.070987 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:27.070989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:27.071003 (+ 14us) server_negotiation.cc:407] Connection header received
0927 04:07:27.071091 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:27.071092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:27.071103 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:27.071153 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 04:07:27.071502 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43286: BlockingRecv error: recv got EOF from 127.0.0.1:43286 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260927 04:07:27.439661 8513 catalog_manager.cc:1259] Loaded cluster ID: 58764aa4a7624cbab0fdb62d6c0d6261
I20260927 04:07:27.439759 8513 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288: loading cluster ID for follower catalog manager: success
I20260927 04:07:27.440544 8513 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288: acquiring CA information for follower catalog manager: success
I20260927 04:07:27.440858 8513 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ec860bd707a49f9a97d52d8fe670288: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:07:27.519243 8558 catalog_manager.cc:1259] Loaded cluster ID: 58764aa4a7624cbab0fdb62d6c0d6261
I20260927 04:07:27.519318 8558 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c: loading cluster ID for follower catalog manager: success
I20260927 04:07:27.520138 8558 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c: acquiring CA information for follower catalog manager: success
I20260927 04:07:27.520471 8558 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b089665079014679aa6a64c4eb3cc03c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 04:07:27.587576 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:27.584827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.17.184.253:46839)
0927 04:07:27.584988 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:27.584990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:27.585000 (+ 10us) server_negotiation.cc:407] Connection header received
0927 04:07:27.585029 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:27.585030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:27.585098 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:27.585182 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:27.585341 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.585981 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.586725 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.586976 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.587034 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:27.587224 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:27.587339 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:27.587415 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:27.587454 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 04:07:27.587484 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":58,"threads_started":1}
I20260927 04:07:27.591955 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:27.589471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.17.184.252:40863)
0927 04:07:27.589624 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:27.589626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:27.589650 (+ 24us) server_negotiation.cc:407] Connection header received
0927 04:07:27.589693 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:27.589694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:27.589729 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:27.589796 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:27.589901 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.590404 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.591215 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.591420 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.591486 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:27.591584 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:27.591691 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:27.591750 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:27.591788 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 04:07:27.591806 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":57,"threads_started":1}
I20260927 04:07:27.597078 8629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:27.593897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.17.184.253:46839)
0927 04:07:27.593948 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:27.593950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:27.594172 (+ 222us) server_negotiation.cc:407] Connection header received
0927 04:07:27.594237 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:27.594238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:27.594248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:27.594325 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:27.594472 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.595270 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.596157 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.596499 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.596593 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:27.596798 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:27.596837 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:27.596888 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:27.596952 (+ 64us) server_negotiation.cc:299] Negotiation successful
0927 04:07:27.596974 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 04:07:27.597393 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:27.593906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.17.184.252:40863)
0927 04:07:27.594088 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:27.594090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:27.594372 (+ 282us) server_negotiation.cc:407] Connection header received
0927 04:07:27.594398 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:27.594399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:27.594410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:27.594452 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:27.594656 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.595398 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.596437 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.596740 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.596824 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:27.596990 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:27.597024 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:27.597069 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:27.597304 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 04:07:27.597328 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260927 04:07:27.597522 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 04:07:27.593761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51798 (local address 127.17.184.254:43669)
0927 04:07:27.593864 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 04:07:27.593867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 04:07:27.593990 (+ 123us) server_negotiation.cc:407] Connection header received
0927 04:07:27.594059 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 04:07:27.594061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 04:07:27.594124 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 04:07:27.594204 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 04:07:27.594381 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.595059 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.596065 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 04:07:27.596326 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 04:07:27.596422 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 04:07:27.596874 (+ 452us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 04:07:27.597004 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 04:07:27.597064 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 04:07:27.597372 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 04:07:27.597401 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":55,"threads_started":1}
I20260927 04:07:27.598924 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 8278
I20260927 04:07:27.604136 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 8353
I20260927 04:07:27.608992 18147 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskJqOnmp/build/release/bin/kudu with pid 8428
2026-09-27T04:07:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2821 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5692 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (274579 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST